https://community.splunk.com/t5/Splunk-Search/how-to-train-Splunk-to-recognize-a-character-set/m-p/48426#M179501 <P>Adding samples to ngram-models simply assists Splunk in guessing a CHARSET that we already support. It cannot be used to add support for a new charset. We have in product support for GB18030, GB231280 and GBK in addition to GB2312.</P> Sun, 12 Sep 2010 04:35:08 GMT Stephen_Sorkin 2010-09-12T04:35:08Z https://community.splunk.com/t5/Splunk-Search/how-to-train-Splunk-to-recognize-a-character-set/m-p/48425#M179500 <P>Hello. My logs contain Simple Chinese characters. After setting CHARSET = GB2312 in the props.conf, some Chinese characters showed up correctly and some didn't. GB2312 encoding is a bit old. GB13000 is the current standard, and it recognizes more characters then GB2312 does. I figure if I can train Splunk to use GB13000 instead of GB2312, it may solve my problem. In the admin manual (http://www.splunk.com/base/Documentation/latest/Admin/Configurecharactersetencoding) it mentions that a sample character set specification file can be added to $SPLUNK_HOME/etc/ngram-models/ to train Splunk to recognize the character set. How do I create such file? Where can I find more information on this topic?</P> <P>Thanks.</P> Sat, 11 Sep 2010 16:13:16 GMT https://community.splunk.com/t5/Splunk-Search/how-to-train-Splunk-to-recognize-a-character-set/m-p/48425#M179500 alextsui 2010-09-11T16:13:16Z https://community.splunk.com/t5/Splunk-Search/how-to-train-Splunk-to-recognize-a-character-set/m-p/48426#M179501 <P>Adding samples to ngram-models simply assists Splunk in guessing a CHARSET that we already support. It cannot be used to add support for a new charset. We have in product support for GB18030, GB231280 and GBK in addition to GB2312.</P> Sun, 12 Sep 2010 04:35:08 GMT https://community.splunk.com/t5/Splunk-Search/how-to-train-Splunk-to-recognize-a-character-set/m-p/48426#M179501 Stephen_Sorkin 2010-09-12T04:35:08Z https://community.splunk.com/t5/Splunk-Search/how-to-train-Splunk-to-recognize-a-character-set/m-p/48427#M179502 <P>Thank you, Stephen.<BR /> I changed the props.conf to CHARSET=GB18030, and the problem was solved.</P> Tue, 14 Sep 2010 14:09:39 GMT https://community.splunk.com/t5/Splunk-Search/how-to-train-Splunk-to-recognize-a-character-set/m-p/48427#M179502 alextsui 2010-09-14T14:09:39Z