topic Re: International character code recognition in Getting Data In

International character code recognition

melonman — Thu, 02 Sep 2010 12:08:07 GMT

Hi there,

I would like to know how to handle international character code in Splunk.

The environment I have here is in Japanese. There are 3 character codes available for Japanese representation, SJIS, EUC and Unicode.

My question is, how Splunk detects the character code of the input, manage the character in index, and handle the characters during search operation.

Thanks!

Re: International character code recognition

gkanapathy — Thu, 02 Sep 2010 23:50:41 GMT

Splunk tries to auto-detect the character set of an input. It may get it wrong. You can specify the character set of an input by setting CHARSET in the props.conf on the input node (i.e., the same server where the inputs.conf is configured), I would recommend specifying it in a [source::...] stanza in that file.

http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding

Re: International character code recognition

melonman — Fri, 03 Sep 2010 08:18:02 GMT

I understand the encoding configuration, but I can't find the explanation how splunk does auto-detect the character set of an input. Is there any information about auto-detect the character set of an input?

Re: International character code recognition

gkanapathy — Mon, 06 Sep 2010 07:46:24 GMT

It guesses based on default trainings it has been given: http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding#If_Splunk_doesn.27t_recognize_a_character_set

Re: International character code recognition

melonman — Thu, 09 Sep 2010 09:15:18 GMT

Thank you for charset related information! well, what I want to know is more like mechanism how Splunk detects char code. Like html or other format, usually charactor codes are specified in the header of the data itself. However, most of case, IT data is simple text and you don't usually know which char code the text is written in. The doc says that splunk does auto-detection, and I want to know how.

Re: International character code recognition

melonman — Tue, 14 Sep 2010 10:30:40 GMT

I will simplify the question. Does splunk use universal encoding detector to detect character encoding of inputs?

Re: International character code recognition

gkanapathy — Tue, 14 Sep 2010 11:10:25 GMT

No. As noted, it uses heuristics based on the training set. As you noted, for log files, there will rarely if ever be encoding indicators, and they will be unreliable.