Dashboards & Visualizations

Windows Security events: XML vs. non-XML format

Communicator

Hi,

We are planning to collect WIndows security events with Splunk. As far as I know, there are two formats: standard and XML with renderXML=1 option.
I've have found some (older) blog/answers questions which says searching with the xml format can be very slow...
So which one should we choose, which format is recommended currently? Pros and cons? Can somebody help me to decide?

Regards,
István

Tags (2)
0 Karma
1 Solution

Ultra Champion

@ikulcsar Personally I favor the normal format over XML - Its easier to read if you are digging through the events.
There are however, some advantages with the XML logs - most notably if you ingest lots of them they are smaller. (Think of your poor licence)
The 'normal' includes lots of boiler plate text which is just noise - we strip this pointless text out thereby shrinking the event size which gives us the best of both worlds.
As some others have noted, sometimes the field extractions can be a bit 'iffy' with XML, but in fairness I have a few servers which randomly start sending double space lines, and that breaks the normal format field extractions too.

I also ran some bench marking on our system and we found that searching 'normal' events was markedly faster than the XML version of the same data - I think I have read posts where people found the opposite.

In short, I don't think there is much in it, but if readability is important to you, stick with the 'normal' format.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

Ultra Champion

@ikulcsar Personally I favor the normal format over XML - Its easier to read if you are digging through the events.
There are however, some advantages with the XML logs - most notably if you ingest lots of them they are smaller. (Think of your poor licence)
The 'normal' includes lots of boiler plate text which is just noise - we strip this pointless text out thereby shrinking the event size which gives us the best of both worlds.
As some others have noted, sometimes the field extractions can be a bit 'iffy' with XML, but in fairness I have a few servers which randomly start sending double space lines, and that breaks the normal format field extractions too.

I also ran some bench marking on our system and we found that searching 'normal' events was markedly faster than the XML version of the same data - I think I have read posts where people found the opposite.

In short, I don't think there is much in it, but if readability is important to you, stick with the 'normal' format.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

Explorer

I have a problem and thought this would be the best post to explain. I currently have a UF installed on a host sending windows security logs to index=wineventlogs in non-XML format (which is what I want). The issue is that the same host is sending the window security logs in xml to index=main, and I cannot find the reason why. I have installed the Splunk App for Win Infra (1.5.0), Win_TA (5.0.1) and Splunk Add on for AD/DNS (1.0.0/1.0.1) on my SH, IDX and UF (where needed).

M guess here is that is something to do with one of the above apps installed, as the UF has the following inputs.conf under the 'etc/app/Splunk_TA_Windows\local\'. No other local files on the UF.

inputs.conf on the UF:
[WinEventLog://Security]
disabled=0
index=wineventlog
renderXml=false
..............(other default data - start_from, current_only etc)

Any tips at this stage would be extremely helpful

0 Karma

Communicator

Hi,
I don't think so this is the right post for this question, but anyway:

  • This is a fresh install (add-ons) or upgraded?
  • Did you try us btool to debug inputs? (eg.: splunk cmd btool inputs list --debug)

Maybe there left some old confs somewhere in the other apps. Be careful, the new Windows add-on didn't create and configure indexes for inputs. This could cause been used the default index. Check the docs, please.

Docs:
https://docs.splunk.com/Documentation/Splunk/7.2.3/Troubleshooting/Usebtooltotroubleshootconfigurati...
https://docs.splunk.com/Documentation/WindowsAddOn/5.0.1/User/Upgrade

Istvan

0 Karma

Explorer

Hi Istvan,

Should I create a separate post? I have checked the inputs and outputs list using btool on the UF and inputs, props and transforms on the IDX, but cant seem to identify the issue.

I'll create another post, thanks

0 Karma

Communicator

Hi,
Yes, please start another post, because it's a separate problem. People can help you easier that way.

(I also try to stop that UF in order to check if there are any rouge UF somewhere with the same hostname.)

Regards,
István

0 Karma

Legend

@ikulcsar, If within Event Viewer you compare the default mode and XML mode, you will notice that XML has all the details while default mode shows you basic information about the event. If your use case does not require you to pull all the details, you can get the default data rather than getting XML which would obviously be faster because lesser details are being indexed per event. However, if your intent of pulling XML Event Viewer Log is to use some of the additional details then you do not have a choice but to turn on renderXML.

Refer to one of my answers regarding the use case for turning on renderXML:
https://answers.splunk.com/answers/550089/setting-up-an-alert-for-computer-booting-in-safe-m.html

You should also see if you can drop unnecessary event logs using WHITELIST and BLACKLIST options in the inputs.conf that would also help.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

New Member

That is an interesting detection, on the day to day use XML events have more issues with extracting CIM fields like src, user,dest app which can hamper more common detections.

0 Karma

New Member

Standard mode is generally preferred for security use case.

0 Karma

SplunkTrust
SplunkTrust

in my experience both work fine.
can you share links to blogs / answers?
will recommend to use the default, (non xml).

hope it helps

0 Karma

Communicator

Hi,

Thanks your reply.

For example:
https://www.batchworks.de/why-using-xml-event-logs-sucks-using-splunk/ or
https://answers.splunk.com/answers/474328/issue-with-windows-xml-security-inputs.html

Both from 2016.. this is why I asked this question and try to get up-to-date information about this topic in general. And I prefer the answers be with explanations if possible, not just a yes/no. I have to build the system, so I should understand it how it works.

Regards,
Istvan

Istvan

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes and swag!