My Windows Event Log size increased dramatically after being forwarded to Splunk. Is there a way to reduce the volume of data received by Splunk?

[cowlthrpdx]

I configured a Windows 2012 server to forward AD event logs to Splunk, everything is working well except for the volume of data being sent to Splunk.

In ~24hr the Windows event log being forwarded increased by ~4GB on disk. The Splunk admin reported that in the same time period the Splunk server received ~70GB from the Windows server.

Other than filtering out events, is there a way to reduce the volume of data received by Splunk?

Thanks,
Rob

[jonmargulies]

This Splunk blog post has a trick that's fully supported and will reduce the size of your Windows events: https://www.splunk.com/blog/2014/11/04/splunk-6-2-feature-overview-xml-event-logs/. The post explains how to send your Windows logs in as XML, which is parsed perfectly by the Windows TA, but the events are a lot smaller because they have less duplicate data (in fact, the XML versions of the events have more useful information stuffed into fewer characters). According to the blog, it reduces event size by about 70%, but in practice it looked like a smaller reduction in my environment.

[kkkelvinkk]

Interesting. However, the following links seems report some different results.
https://www.batchworks.de/why-using-xml-event-logs-sucks-using-splunk/

I will give a try tommorow.

[gfreitas]

Is that a new server? Maybe you can add ignoreOlderThan = 24h on inputs.conf to make sure this data is really from today.