Archive

Splunk indexes text file in binary format ?

Contributor

Hello,

When i monitored a file , at first its content is forwarded from forwarder to indexer in text format, so i can make a table with that content.

But after the system has updated that file by deleting it and creating a new same-name file with different content, I see that Splunk indexes its new data in binary format

5:21:47.000 PM

\x001\x002\x00/\x001\x002\x00/\x001\x002\x00 \x001\x008\x00:\x000\x005\x00:\x001\x007\x00,\x00 \x000\x000\x003\x002\x009\x00 \x00[\x000\x00x\x000\x007\x00E\x004\x00]\x00 \x00=\x00>\x00[\x00T\x00R\x00A\x00N\x00S\x00A\x00C\x00T\x00I\x00O\x00N\x00I\x00N\x00F\x00O\x00

However, i can open this file in notepad and view its content without any issue. So can you tell me what is the problem i have got ?

Tags (1)
0 Karma
1 Solution

Contributor

Yeah, it's worked. This charset has to be set on HF. Thanks

View solution in original post

0 Karma

Contributor

Yeah, it's worked. This charset has to be set on HF. Thanks

View solution in original post

0 Karma

Contributor

I think, If you open a file in the forwarder, it will create .swp file in the same folder, as and when .swp file is created it will be forwarded to indexer for indexing. Thats why you will see that binary format data and also you can set charset as HF.

0 Karma

SplunkTrust
SplunkTrust

I'd guess you have to set charset in the HF as well. That's where the real "cooking" part of the indexing process is occurring.

0 Karma

Contributor

Yeah, data is forwarded by the following order:UniversalForwarder -> HeavyForwarder -> Indexer , i think it would be right setting charset in indexer, but iam still getting that issue

0 Karma

SplunkTrust
SplunkTrust

I'd contact Splunk Support. I think some data input config somewhere needs to be configured to tell Splunk that the incoming data is UTF-16, otherwise it always assumes everything is UTF-8, which explains what you're seeing. Possibly Firefox is doing some overly clever detection and "fixing" the situation at the browser level, but there's still a fundamental problem in the middle layers.

0 Karma

Contributor

It's so strange. When i viewed search results in indexer on Firefox, the data seemed to be well displayed, but with Chrome it showed like "x001x002x00/x001x002x00/x0.....", but in 2 cases, i cannot use any report command to create table from them. The data's encoding is UTF-16LE.

0 Karma

SplunkTrust
SplunkTrust

Interesting that if you remove all the x00's, that sample ends with "[TRANSACTION]". I've seen something like this before. Is it possible this is a single-byte vs double-byte issue, or a Unicode/UTF8/UTF16 issue?

0 Karma