Getting Data In

International character code recognition

Motivator

Hi there,

I would like to know how to handle international character code in Splunk.

The environment I have here is in Japanese. There are 3 character codes available for Japanese representation, SJIS, EUC and Unicode.

My question is, how Splunk detects the character code of the input, manage the character in index, and handle the characters during search operation.

Thanks!

0 Karma
1 Solution

Splunk Employee
Splunk Employee

Splunk tries to auto-detect the character set of an input. It may get it wrong. You can specify the character set of an input by setting CHARSET in the props.conf on the input node (i.e., the same server where the inputs.conf is configured), I would recommend specifying it in a [source::...] stanza in that file.

http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding

View solution in original post

Splunk Employee
Splunk Employee

Splunk tries to auto-detect the character set of an input. It may get it wrong. You can specify the character set of an input by setting CHARSET in the props.conf on the input node (i.e., the same server where the inputs.conf is configured), I would recommend specifying it in a [source::...] stanza in that file.

http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding

View solution in original post

Splunk Employee
Splunk Employee

No. As noted, it uses heuristics based on the training set. As you noted, for log files, there will rarely if ever be encoding indicators, and they will be unreliable.

0 Karma

Motivator

I will simplify the question. Does splunk use universal encoding detector to detect character encoding of inputs?

0 Karma

Motivator

Thank you for charset related information! well, what I want to know is more like mechanism how Splunk detects char code. Like html or other format, usually charactor codes are specified in the header of the data itself. However, most of case, IT data is simple text and you don't usually know which char code the text is written in. The doc says that splunk does auto-detection, and I want to know how.

0 Karma

Splunk Employee
Splunk Employee
0 Karma

Motivator

I understand the encoding configuration, but I can't find the explanation how splunk does auto-detect the character set of an input. Is there any information about auto-detect the character set of an input?

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!