Why are indexers not parsing events?

[devin07]

Fairly new to Splunk so may not have the correct terms for everything. Currently working in a distributed environment with Splunk Enterprise with windows and Linux host. These hosts are sending logs via UFs to the clustered indexers. There is also an HF that is receiving logs from apps and AWS. My issue is that the logs coming from my UF are not being parsed into field name-value pairs. The windows/Linux host, indexers, and Search Heads all have the splunk_TA_nix and splunk_TA_windows add-ons installed.  I almost feel like my indexers are not parsing the data that is coming in.

Log data is getting into Splunk and I can see my events however it is all in a format similar to this, very crude I know.

<data><data><data>1039<data><data><data>time<data><data>program<data>splunk<data>

I would like it to be in field name values.  At some point I was receiving logs in this format however I am no longer. What could be causing this? 

time: 10:39
program: splunk

[gcusello]

Hi @devin07,

one question: do logs from UFs pass throgh the HF?

if yes, you have to install windows and linux TA also on HF.

I suppose that you are using the Linux and windows TAs also on UFs to input data, is it correct?

Ciao.

Giuseppe

[devin07]

Hey, @gcusello  Thanks for responding, so no the UFs do not send logs to the HF in our env. The UFs send logs straight to the indexers. We are using the HF for AWS cloud trail/cloudwatch logs and for some applications. 

The Linux and windows TAs on the UF are used to input data, yes.  Data is getting into Splunk fine, it's just not being parsed it seems if that makes since

[gcusello]

Hi @devin07,

I suppose that you're using the same TA both on UFs and IDXs, if not check the sourcetype assigned on UFs.

Ciao.

Giuseppe

[devin07]

Same source type, I used a DS to deploy the TA and just checked the source types are the same

[devin07]

Not sure if this helps I was looking further and under index=windeventlog I am only seeing a source type of XmlWinEventLog and not just a source type of WinEventLog our inputs.conf does have renderxml=true. So if it helps it is looking like the XmlWinEventLog source types are not being parsed correctly. 

Could it be that my fields are being extracted automatically?

[gcusello · Accepted solution]

Hi @devin07,

are you logs in XML or raw?

try using 

renderxml=false

Ciao.

Giuseppe

[devin07]

Thanks @gcusello This was it, we had used a file that had this set to true rather than false thank you!!!

[gcusello]

Hi @devin07 ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉