topic Re: Splunk re-indexing rolled over log file causing duplicate (two) copies of data in Getting Data In

Splunk re-indexing rolled over log file causing duplicate (two) copies of data

sfmandmdev — Mon, 24 Sep 2012 21:25:39 GMT

We have a log file rotation policy that rolls over based on size (64MB). For some reason, every now and then (frequent but not all the time), splunk forwarder thinks the rolled over file is a new file and ships it again causing duplicates in the indexer.

We would find the same event from filenames blah and blah.0 (rolledover file name).

Any clues what might be causing this issue?

Re: Splunk re-indexing rolled over log file causing duplicate (two) copies of data

Ayn — Mon, 24 Sep 2012 21:59:24 GMT

Do you have any particular crcSalt settings set in inputs.conf for this particular source?

Re: Splunk re-indexing rolled over log file causing duplicate (two) copies of data

sfmandmdev — Tue, 25 Sep 2012 18:22:03 GMT

We don't have any crcSalt settings set. Also this does not happen all the time i.e All rolled over versions of the same log file are not duplicated.

Re: Splunk re-indexing rolled over log file causing duplicate (two) copies of data

dwaddle — Tue, 25 Sep 2012 20:18:52 GMT

How are you performing your roll-over? It is a rename or a copy?

Re: Splunk re-indexing rolled over log file causing duplicate (two) copies of data

hmahendrakumar — Tue, 25 Sep 2012 20:22:52 GMT

It is a rename.
Note: ZFS is underlying FS
Each process has a lot of threads that
write to log files protected by a mutex. So only one thread can write at a time.
When we see the file growing to exceed this size (~64MB), we acquire the mutex
blocking any writes to the file, closing the file, deleting oldest generation N (BLAHFILENAME.N)
then for (n = 0; n < N; n++) rename BLAHFILENAME.N to BLAHFILENAME.N+1
then finally renaming current log file BLAHFILENAME to BLAHFILENAME.0
then creating a new empty log file BLAHFILENAME and releasing the mutex
allowing all threads to write to the new file.