topic Re: Ingestion of Windows events works correctly in the classic WinEventLog format but not in the XmlWinEventLog format in Splunk Enterprise

Ingestion of Windows events works correctly in the classic WinEventLog format but not in the XmlWinEventLog format

MCH2018 — Fri, 01 Dec 2023 14:14:02 GMT

Hello Team,

I got a weird issue, that I struggle to troubleshoot.

A month ago, I realized that my WinEventLog logs were consuming too much of my licenses, so I decided to index them in the XmlWinEventLog format. To do this, I simply modified the inputs.conf file of my Universal Forwarder.

I changed from this configuration :

[WinEventLog://Security] 
disabled = 0 
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\sgroupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\sgroupPolicyContainer)"
renderXml = false
sourcetype = WinEventLog
index = wineventlog

To this configuration:

[WinEventLog://Security] 
disabled = 0 
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\sgroupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\sgroupPolicyContainer)"
renderXml = true
sourcetype = XmlWinEventLog
index = wineventlog

Then I started receiving events and my license usage reduced, which made me happy. However, upon closer observation, I realized that I wasn't receiving all the events as before. Indeed, I now observe that the event frequency of the XmlWinEventLog logs is random.

You can observe this on these timelines :

And in the metrics :

On the other hand, with the WinEventLog format, I have no issues:

I tried reinstalling the UF, there are no interesting errors in the splunkd.log, and I am out of ideas for troubleshooting.
Thank you for your help.

Re: Ingestion of Windows events works correctly in the classic WinEventLog format but not in the XmlWinEventLog format

azteksites — Sat, 02 Dec 2023 15:01:50 GMT

Your blacklist regex expressions may not be compatible with with the XML format for your indexed events.

Referenced from https://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowseventlogdata#Use_blacklists_and_whitelists_to_filter_on_XML-based_events :

Render event data as extensible markup language (XML) supplied by the Windows Event Log subsystem. This setting is optional.

A value of 1 or true means to render the events as XML. A value of 0 or false means to render the events as plain text.

If you set renderXml to true, and if you want to also create allow lists or deny lists to filter event data, you must use the $XmlRegex special key in your allow lists or deny lists.

0 (false)

Re: Ingestion of Windows events works correctly in the classic WinEventLog format but not in the XmlWinEventLog format

MCH2018 — Tue, 05 Dec 2023 10:00:28 GMT

Thanks for your help, I haven't been able to test your solution yet.
I'm supposed to do it this week, so I'll get back to you.

Re: Ingestion of Windows events works correctly in the classic WinEventLog format but not in the XmlWinEventLog format

PickleRick — Tue, 05 Dec 2023 13:22:03 GMT

While the blacklist format might not be compatible with the XML event format, that should not cause decrease of the number of events, quite the contrary.

I'd check firstly whether your overall number of events (not just bursts) indeed did decrease. In other words - are you indeed losing events or are are they by any chance getting "choked" but finally get through in shorter but higher-thruput bursts.

Re: Ingestion of Windows events works correctly in the classic WinEventLog format but not in the XmlWinEventLog format

MCH2018 — Tue, 05 Dec 2023 14:22:36 GMT

That was one of my theories, but unfortunately, after checking, we do have some missing events.

We only receive random events in XML and all events in wineventlog format.

Re: Ingestion of Windows events works correctly in the classic WinEventLog format but not in the XmlWinEventLog format

MCH2018 — Fri, 08 Dec 2023 09:19:10 GMT

Hello,

Unfortunatly, this solution doesn't solve anything.

Re: Ingestion of Windows events works correctly in the classic WinEventLog format but not in the XmlWinEventLog format

PickleRick — Sat, 09 Dec 2023 09:20:05 GMT

The thing I could suggest is enabling debug and trying to look into forwarder's logs but that's a long shot and I have really no concrete advice what to look for. Kinda like "exploratory surgery".