HI Experts
I am having an issue in indexing the log file which gets rotated ever hour. The log file error.log gets rotated every hour at top of the hour and a new file is created with the same name(error.log). The old file gets renamed and zipped to error.log._timestamp.gz.
Sometimes splunk does not index the file for an hour and resumes the indexing once the file is again rotated so the complete 1 hour logs gets skipped. Before splunk resumes the indexing following error message is logged.
WatchedFile - Checksum for seekptr didn't match, will re-read entire file
Every file has a different content because each event has a timestamp so first 256 characters should not much the fishbucket.
[monitor:///data/logs/]
_TCP_ROUTING = indexers123
index = indexname
sourcetype = sourcetypename
initCrcLength = 1024
whitelist=\.log$
disabled = 0
Splunk version is 7.0.3
food for thoughts
Have you thought monitoring the rotated file too ?
Splunk had a logic to avoid reindexing the file
Sometimes it helps if the last events of the file were not indexed before the file rotated.
if the problem is during the rotation process, maybe disable the monitoring before the rotation and re-enable after.