International character code recognition

[melonman]

Hi there,

I would like to know how to handle international character code in Splunk.

The environment I have here is in Japanese. There are 3 character codes available for Japanese representation, SJIS, EUC and Unicode.

My question is, how Splunk detects the character code of the input, manage the character in index, and handle the characters during search operation.

Thanks!

[gkanapathy · Accepted solution]

Splunk tries to auto-detect the character set of an input. It may get it wrong. You can specify the character set of an input by setting CHARSET in the props.conf on the input node (i.e., the same server where the inputs.conf is configured), I would recommend specifying it in a [source::...] stanza in that file.

http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding

[gkanapathy]

No. As noted, it uses heuristics based on the training set. As you noted, for log files, there will rarely if ever be encoding indicators, and they will be unreliable.

[melonman]

I will simplify the question. Does splunk use universal encoding detector to detect character encoding of inputs?

[melonman]

Thank you for charset related information! well, what I want to know is more like mechanism how Splunk detects char code. Like html or other format, usually charactor codes are specified in the header of the data itself. However, most of case, IT data is simple text and you don't usually know which char code the text is written in. The doc says that splunk does auto-detection, and I want to know how.

[gkanapathy]

It guesses based on default trainings it has been given: http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding#If_Splunk_doesn.2...

[melonman]

I understand the encoding configuration, but I can't find the explanation how splunk does auto-detect the character set of an input. Is there any information about auto-detect the character set of an input?