How to check for files that are indexed multiple times

[lakshman237]

Greetings.,

I have universal forwarders reading files from a common directory, with whitelists. I have reviewed the monitor stanza and couldn't see the likelyhood of samefile being indexed twice.

Is there a way to find same files being indexed more than once from the indexer?

thanks
laks

[kristian_kolb]

Depending on how unique your messages are, you could always try

...| stats count last(source) AS dup_file by _raw | search count > 1 | table dup_file 

if you get any results, you will likely have indexed the same event/file more than once.
The results will be rather simplistic, but...it might help.

_raw is the whole event, including its timestamp, which should be fairly unique in combination with the rest of the event text.

Please not that it may take some time, since splunk will have to go through all events. Some further qualification (sourcetype, host etc) may be necessary to speed things up.

/k