Splunk Enterprise

duplicate events, multiple indexed times

willsy
Path Finder

hello, 

i am monitoring windows event logs and ingesting them to my indexers, the issue is that even with a unique EventRecordID i am seeing multiple events in Splunk, sometimes up to 28. 

second to that, when i complete the two searches in the picture i can see that the same event is being indexed multiple times (14) between 13:33:31 and 13:36:00

any help on how to rectify this issue is greatly appreciated. 

please see attached the two searches showing multiple indexed results and also multiple indexed times. 

Paste 1.PNGpaste 2.PNG

Labels (2)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

Are your log files being rotated, such as to zip files? Duplicate events such as this can happen if log files are rotated, renamed, or compressed. Splunk will see those as new files and ingest them, unless you explicitly blacklist them.

Try adding source to your search to see what files the events are coming from.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

mani1
New Member

How to resolve the issue if logs are rotated or compressed. How to blacklist them? 

Splunk is ingesting duplicate events in my org.

 

Help me how to fix issue if logs are rotated or compressed.

Highly appreciate your help.

0 Karma

willsy
Path Finder

Hey codebuilder, 

the source is in the second pic, it shows all source and sourcetypes are the same for all dupicates. 

i am using the standard windows app, there is no log rotating or zip as it is purely the windows events being monitored. 

0 Karma

codebuilder
SplunkTrust
SplunkTrust

That pic doesn't include the actual source file.

Try running the following:

|tstats count where index=windows_event_log by source

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

willsy
Path Finder

Good morning, 

So if i did your search without specifically asking for that EventRecordID i have the three sources 

XmlWinEventLog:Application
XmlWinEvenLog:Security
XmlWinEventLog:System

If however i do the same with that specific EventRecordID it is only aligned to source;
XmlWinEventLog:Security

Which means that it is coming from only one source. 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...