Reducing License usage by eliminating duplicate events from WIndows

[gba8912]

Hello,

I am looking for a way to reduce our license usage by eliminating duplicate events being forwarded from a windows DC. For example, event id 4624 (successful logon) generates a handful of events in windows for every logon and all of those are being sent to Splunk. Is there a regex we can use in the input.conf file to only allow one event to be logged maybe based on the timestamp or something? Here is what we are currently using:

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 0
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"
renderXml=false
index = wineventlog
ignoreOlderThan = 2d
current_only=1
whitelist = EventCode = "4624"
blacklist1 = EventCode="4624" Message=".*[\S\s]*Account\sName:\s+[\S+]+[\$]"

thanks!

[gcusello · Accepted solution]

Hi @gba8912,

you cannot remove duplicated events before indexing because to know that an event is already indexed you have to index it!

If you want to reduce your license consuption, at first, you can disable perfmon that are very verbose and reduce the execution of scripts.

Then run a simple search "index=wineventlog | stats count BY EventCode | sort -count" and identify the most present events, then identify the ones you don't need and then filter them following the steps at https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Routeandfilterdatad#Filter_event_data_....

Ciao.

Giuseppe