topic charset issue in Getting Data In

charset issue

perlish — Mon, 29 Apr 2013 13:23:33 GMT

Hi, everybody.

I want use splunk to index the data which contain chinese.

Firstly, the base data will send to my splunk universal forwarder.

Then,my universal forwarder will forward the data to my splunk.

The base data char is gb2312.

This is the inputs.conf in universal forwarder.

[udp://514]
connection_host = none
sourcetype = businesslog

This is the props.conf in splunk.

[businesslog]
CHARSET = GB2312

But now the chinese still can`t display correctly.

Wheather i need to define the props.conf in universal forwarder?

Thanks.

Re: charset issue

woodcock — Wed, 13 May 2015 06:58:59 GMT

This props.conf setting is an index-time setting so it needs to be certain places depending on your configuration. If you are using Heavy Forwarders, it must be on your forwarders but if you are using Universal or Light-Weight Forwarders, it must be on (all of) your Indexers; is it on all of your Indexers?

Re: charset issue

acharlieh — Mon, 18 May 2015 13:18:42 GMT

CHARSET is an attribute set into the pipeline data at input time, the props.conf should be on your Universal Forwarders. References:

Re: charset issue

acharlieh — Mon, 18 May 2015 13:22:55 GMT

It's checked at parse time, however it's an input time setting. Otherwise the UF would not be able to know if it was cutting multibyte characters in half or not as it sends chunks of data on to the parser.

Re: charset issue

woodcock — Mon, 18 May 2015 14:59:11 GMT

Fair enough but my point stands that this props.conf should be exported to every node that needs to modify the data (forwarders and indexers) and my suspicion is that it has not been so distributed.