Getting Data In

How to check for files that are indexed multiple times

Path Finder


I have universal forwarders reading files from a common directory, with whitelists. I have reviewed the monitor stanza and couldn't see the likelyhood of samefile being indexed twice.

Is there a way to find same files being indexed more than once from the indexer?


Tags (2)
0 Karma

Ultra Champion

Depending on how unique your messages are, you could always try

...| stats count last(source) AS dup_file by _raw | search count > 1 | table dup_file 

if you get any results, you will likely have indexed the same event/file more than once.
The results will be rather simplistic, might help.

_raw is the whole event, including its timestamp, which should be fairly unique in combination with the rest of the event text.

Please not that it may take some time, since splunk will have to go through all events. Some further qualification (sourcetype, host etc) may be necessary to speed things up.


.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!