Splunk indexes some events several times?

[cfortune]

In some of our indexed logs, I'll see several log entries for the same log at the same time. I thought this may be an issue with the log file itself but I just checked and sure enough, there aren't several of the same event from the same time in there. Any idea what could be causing this?

Here is a screen shot.

The number 11196 is a unique id associated with that particular log in. As you can see, it's the same for all 10 results shown. If someone really logged in that fast, they'd have different id's there. There is an 8 hour timezone offset applied to these logs as well. Don't know if that has anything to do with it.

Also, these logs are indexed via a cifs mount on the Splunk server.

[gkanapathy]

There are applications that, in addition to adding log entries to the end of the file, will also rewrite some data at the beginning. If yours does this, then each time it writes, it will make Splunk think the entire file is new (since the beginning of the file has changed) and the entire file will be re-indexed.

[dwaddle]

Try putting a Splunk forwarder on the machine which holds this data, and let the forwarder read the file locally and then forward the events on to the indexer.

I don't use CIFS, but I have seen some similarly odd events indexing over NFS - the application's buffer flushing behavior combined with the timing of the NFS reads would cause some odd behaviors that we eliminated using a forwarder.