I am using logrotate to rotate my files, with the option copytruncate.
http://linuxcommand.org/man_pages/logrotate8.html
And It is causing splunk to reindex the file during the rotation.
see my rotate script /etc/logrotate.d/test
"/home/feed/test.log" {
rotate 5
size=50M
sharedscripts
copytruncate
compress
}
What happens is :
- the rotation creates a new rotated file. (test.log.1)
- the rotation copy the content of the file to the new file (from line 1 to line X), then compress it.
- the rotation then truncates the original file by removing the first lines per chunks, until reaching the line X.
At the same time, splunk is monitoring the file, and check the first 256 chars for the crc.
- during the truncate, because the first lines are changing, splunk redetect the file as new, and reindex it all.
- we find duplicates of the lines, the first lines with fewer duplicates, the last lines with more duplicates.
How to identify the issue :
- look for duplicates and the time when they were generated. (not the event timestamp)
source=path/to/my/logfile | convert ctime(_indextime) AS indextime| stats count values(indextime) AS indextime by _raw | where count >1
- at the time of the rotation, look on the forwarder splunkd.log, check for multiple events like :
11-05-2014 10:48:33.924 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/home/feed/test.log
11-05-2014 10:48:34.234 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/home/feed/test.log
etc...
