When i monitored a file , at first its content is forwarded from forwarder to indexer in text format, so i can make a table with that content.
But after the system has updated that file by deleting it and creating a new same-name file with different content, I see that Splunk indexes its new data in binary format
\x001\x002\x00/\x001\x002\x00/\x001\x002\x00 \x001\x008\x00:\x000\x005\x00:\x001\x007\x00,\x00 \x000\x000\x003\x002\x009\x00 \x00[\x000\x00x\x000\x007\x00E\x004\x00]\x00 \x00=\x00>\x00[\x00T\x00R\x00A\x00N\x00S\x00A\x00C\x00T\x00I\x00O\x00N\x00I\x00N\x00F\x00O\x00
However, i can open this file in notepad and view its content without any issue. So can you tell me what is the problem i have got ?
I think, If you open a file in the forwarder, it will create .swp file in the same folder, as and when .swp file is created it will be forwarded to indexer for indexing. Thats why you will see that binary format data and also you can set charset as HF.
Yeah, data is forwarded by the following order:UniversalForwarder -> HeavyForwarder -> Indexer , i think it would be right setting charset in indexer, but iam still getting that issue
I'd contact Splunk Support. I think some data input config somewhere needs to be configured to tell Splunk that the incoming data is UTF-16, otherwise it always assumes everything is UTF-8, which explains what you're seeing. Possibly Firefox is doing some overly clever detection and "fixing" the situation at the browser level, but there's still a fundamental problem in the middle layers.
It's so strange. When i viewed search results in indexer on Firefox, the data seemed to be well displayed, but with Chrome it showed like "x001x002x00/x001x002x00/x0.....", but in 2 cases, i cannot use any report command to create table from them. The data's encoding is UTF-16LE.
Interesting that if you remove all the x00's, that sample ends with "[TRANSACTION]". I've seen something like this before. Is it possible this is a single-byte vs double-byte issue, or a Unicode/UTF8/UTF16 issue?