Splunk Search

Splunk indexes text file in binary format ?

sieutruc
Contributor

Hello,

When i monitored a file , at first its content is forwarded from forwarder to indexer in text format, so i can make a table with that content.

But after the system has updated that file by deleting it and creating a new same-name file with different content, I see that Splunk indexes its new data in binary format

5:21:47.000 PM

\x001\x002\x00/\x001\x002\x00/\x001\x002\x00 \x001\x008\x00:\x000\x005\x00:\x001\x007\x00,\x00 \x000\x000\x003\x002\x009\x00 \x00[\x000\x00x\x000\x007\x00E\x004\x00]\x00 \x00=\x00>\x00[\x00T\x00R\x00A\x00N\x00S\x00A\x00C\x00T\x00I\x00O\x00N\x00I\x00N\x00F\x00O\x00

However, i can open this file in notepad and view its content without any issue. So can you tell me what is the problem i have got ?

Tags (1)
0 Karma
1 Solution

sieutruc
Contributor

Yeah, it's worked. This charset has to be set on HF. Thanks

View solution in original post

0 Karma

sieutruc
Contributor

Yeah, it's worked. This charset has to be set on HF. Thanks

0 Karma

srinathd
Contributor

I think, If you open a file in the forwarder, it will create .swp file in the same folder, as and when .swp file is created it will be forwarded to indexer for indexing. Thats why you will see that binary format data and also you can set charset as HF.

0 Karma

sideview
SplunkTrust
SplunkTrust

I'd guess you have to set charset in the HF as well. That's where the real "cooking" part of the indexing process is occurring.

0 Karma

sieutruc
Contributor

Yeah, data is forwarded by the following order:UniversalForwarder -> HeavyForwarder -> Indexer , i think it would be right setting charset in indexer, but iam still getting that issue

0 Karma

sideview
SplunkTrust
SplunkTrust

I'd contact Splunk Support. I think some data input config somewhere needs to be configured to tell Splunk that the incoming data is UTF-16, otherwise it always assumes everything is UTF-8, which explains what you're seeing. Possibly Firefox is doing some overly clever detection and "fixing" the situation at the browser level, but there's still a fundamental problem in the middle layers.

0 Karma

sieutruc
Contributor

It's so strange. When i viewed search results in indexer on Firefox, the data seemed to be well displayed, but with Chrome it showed like "x001x002x00/x001x002x00/x0.....", but in 2 cases, i cannot use any report command to create table from them. The data's encoding is UTF-16LE.

0 Karma

sideview
SplunkTrust
SplunkTrust

Interesting that if you remove all the x00's, that sample ends with "[TRANSACTION]". I've seen something like this before. Is it possible this is a single-byte vs double-byte issue, or a Unicode/UTF8/UTF16 issue?

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...