Reducing License usage by eliminating duplicate events from WIndows

gba8912 — Thu, 19 Nov 2020 21:26:36 GMT

Hello,

I am looking for a way to reduce our license usage by eliminating duplicate events being forwarded from a windows DC. For example, event id 4624 (successful logon) generates a handful of events in windows for every logon and all of those are being sent to Splunk. Is there a regex we can use in the input.conf file to only allow one event to be logged maybe based on the timestamp or something? Here is what we are currently using:

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 0
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"
renderXml=false
index = wineventlog
ignoreOlderThan = 2d
current_only=1
whitelist = EventCode = "4624"
blacklist1 = EventCode="4624" Message=".*[\S\s]*Account\sName:\s+[\S+]+[\$]"

thanks!

Re: Reducing License usage by eliminating duplicate events from WIndows

gcusello — Fri, 20 Nov 2020 07:32:27 GMT

Hi @gba8912,

you cannot remove duplicated events before indexing because to know that an event is already indexed you have to index it!

If you want to reduce your license consuption, at first, you can disable perfmon that are very verbose and reduce the execution of scripts.

Then run a simple search "index=wineventlog | stats count BY EventCode | sort -count" and identify the most present events, then identify the ones you don't need and then filter them following the steps at https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues.

Ciao.

Giuseppe

topic Reducing License usage by eliminating duplicate events from WIndows in Getting Data In

Reducing License usage by eliminating duplicate events from WIndows

Re: Reducing License usage by eliminating duplicate events from WIndows