topic Re: duplicate events, multiple indexed times in Splunk Enterprise

duplicate events, multiple indexed times

willsy — Wed, 14 Jul 2021 14:19:34 GMT

hello,

i am monitoring windows event logs and ingesting them to my indexers, the issue is that even with a unique EventRecordID i am seeing multiple events in Splunk, sometimes up to 28.

second to that, when i complete the two searches in the picture i can see that the same event is being indexed multiple times (14) between 13:33:31 and 13:36:00

any help on how to rectify this issue is greatly appreciated.

please see attached the two searches showing multiple indexed results and also multiple indexed times.

Re: duplicate events, multiple indexed times

codebuilder — Wed, 14 Jul 2021 19:48:29 GMT

Are your log files being rotated, such as to zip files? Duplicate events such as this can happen if log files are rotated, renamed, or compressed. Splunk will see those as new files and ingest them, unless you explicitly blacklist them.

Try adding source to your search to see what files the events are coming from.

Re: duplicate events, multiple indexed times

willsy — Wed, 14 Jul 2021 19:51:58 GMT

Hey codebuilder,

the source is in the second pic, it shows all source and sourcetypes are the same for all dupicates.

i am using the standard windows app, there is no log rotating or zip as it is purely the windows events being monitored.

Re: duplicate events, multiple indexed times

codebuilder — Wed, 14 Jul 2021 20:08:45 GMT

That pic doesn't include the actual source file.

Try running the following:

|tstats count where index=windows_event_log by source

Re: duplicate events, multiple indexed times

willsy — Thu, 15 Jul 2021 07:32:33 GMT

Good morning,

So if i did your search without specifically asking for that EventRecordID i have the three sources

XmlWinEventLog:Application
XmlWinEvenLog:Security
XmlWinEventLog:System

If however i do the same with that specific EventRecordID it is only aligned to source;
XmlWinEventLog:Security

Which means that it is coming from only one source.

Re: duplicate events, multiple indexed times

mani1 — Tue, 11 Jan 2022 17:55:16 GMT

How to resolve the issue if logs are rotated or compressed. How to blacklist them?

Splunk is ingesting duplicate events in my org.

Help me how to fix issue if logs are rotated or compressed.

Highly appreciate your help.