Splunk indexes text file in binary format ?

[sieutruc]

Hello,

When i monitored a file , at first its content is forwarded from forwarder to indexer in text format, so i can make a table with that content.

But after the system has updated that file by deleting it and creating a new same-name file with different content, I see that Splunk indexes its new data in binary format

5:21:47.000 PM

\x001\x002\x00/\x001\x002\x00/\x001\x002\x00 \x001\x008\x00:\x000\x005\x00:\x001\x007\x00,\x00 \x000\x000\x003\x002\x009\x00 \x00[\x000\x00x\x000\x007\x00E\x004\x00]\x00 \x00=\x00>\x00[\x00T\x00R\x00A\x00N\x00S\x00A\x00C\x00T\x00I\x00O\x00N\x00I\x00N\x00F\x00O\x00

However, i can open this file in notepad and view its content without any issue. So can you tell me what is the problem i have got ?

[sieutruc · Accepted solution]

Yeah, it's worked. This charset has to be set on HF. Thanks

[srinathd]

I think, If you open a file in the forwarder, it will create .swp file in the same folder, as and when .swp file is created it will be forwarded to indexer for indexing. Thats why you will see that binary format data and also you can set charset as HF.

[sideview]

I'd guess you have to set charset in the HF as well. That's where the real "cooking" part of the indexing process is occurring.

[sieutruc]

Yeah, data is forwarded by the following order:UniversalForwarder -> HeavyForwarder -> Indexer , i think it would be right setting charset in indexer, but iam still getting that issue

[sideview]

I'd contact Splunk Support. I think some data input config somewhere needs to be configured to tell Splunk that the incoming data is UTF-16, otherwise it always assumes everything is UTF-8, which explains what you're seeing. Possibly Firefox is doing some overly clever detection and "fixing" the situation at the browser level, but there's still a fundamental problem in the middle layers.

[sieutruc]

It's so strange. When i viewed search results in indexer on Firefox, the data seemed to be well displayed, but with Chrome it showed like "x001x002x00/x001x002x00/x0.....", but in 2 cases, i cannot use any report command to create table from them. The data's encoding is UTF-16LE.

[sideview]

Interesting that if you remove all the x00's, that sample ends with "[TRANSACTION]". I've seen something like this before. Is it possible this is a single-byte vs double-byte issue, or a Unicode/UTF8/UTF16 issue?