Splunk Enterprise

Ingestion of Windows events works correctly in the classic WinEventLog format but not in the XmlWinEventLog format

MCH2018
Explorer

Hello Team,

I got a weird issue, that I struggle to troubleshoot.

A month ago, I realized that my WinEventLog logs were consuming too much of my licenses, so I decided to index them in the XmlWinEventLog format. To do this, I simply modified the inputs.conf file of my Universal Forwarder.

I changed from this configuration :

[WinEventLog://Security] 
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\sgroupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\sgroupPolicyContainer)"
renderXml = false
sourcetype = WinEventLog
index = wineventlog

To this configuration:

[WinEventLog://Security] 
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\sgroupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\sgroupPolicyContainer)"
renderXml = true
sourcetype = XmlWinEventLog
index = wineventlog

Then I started receiving events and my license usage reduced, which made me happy. However, upon closer observation, I realized that I wasn't receiving all the events as before. Indeed, I now observe that the event frequency of the XmlWinEventLog logs is random.

You can observe this on these timelines :

MCH2018_0-1701439929580.png

 

And in the metrics :

MCH2018_1-1701439945037.png

 

On the other hand, with the WinEventLog format, I have no issues:

MCH2018_2-1701439959223.png

 

I tried reinstalling the UF, there are no interesting errors in the splunkd.log, and I am out of ideas for troubleshooting.
Thank you for your help.

Labels (2)
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

While the blacklist format might not be compatible with the XML event format, that should not cause decrease of the number of events, quite the contrary.

I'd check firstly whether your overall number of events (not just bursts) indeed did decrease. In other words - are you indeed losing events or are are they by any chance getting "choked" but finally get through in shorter but higher-thruput bursts.

 

0 Karma

MCH2018
Explorer

That was one of my theories, but unfortunately, after checking, we do have some missing events.

We only receive random events in XML and all events in wineventlog format.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The thing I could suggest is enabling debug and trying to look into forwarder's logs but that's a long shot and I have really no concrete advice what to look for. Kinda like "exploratory surgery".

0 Karma

azteksites
Explorer

 Your blacklist regex expressions may not be compatible with with the XML format for your indexed events.

Referenced from https://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowseventlogdata#Use_blacklists_a...

Render event data as extensible markup language (XML) supplied by the Windows Event Log subsystem. This setting is optional.

A value of 1 or true means to render the events as XML. A value of 0 or false means to render the events as plain text.

If you set renderXml to true, and if you want to also create allow lists or deny lists to filter event data, you must use the $XmlRegex special key in your allow lists or deny lists.

0 (false)
0 Karma

MCH2018
Explorer

Thanks for your help, I haven't been able to test your solution yet.
I'm supposed to do it this week, so I'll get back to you.

0 Karma

MCH2018
Explorer

Hello,

Unfortunatly, this solution doesn't solve anything.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...