Deployment Architecture

Splunk forwarder Skipping Log files occasionally

vaibhavagg2006
Communicator

HI Experts
I am having an issue in indexing the log file which gets rotated ever hour. The log file error.log gets rotated every hour at top of the hour and a new file is created with the same name(error.log). The old file gets renamed and zipped to error.log._timestamp.gz.
Sometimes splunk does not index the file for an hour and resumes the indexing once the file is again rotated so the complete 1 hour logs gets skipped. Before splunk resumes the indexing following error message is logged.

WatchedFile - Checksum for seekptr didn't match, will re-read entire file

Every file has a different content because each event has a timestamp so first 256 characters should not much the fishbucket.

[monitor:///data/logs/]
_TCP_ROUTING = indexers123
index = indexname
sourcetype = sourcetypename
initCrcLength = 1024
whitelist=\.log$
disabled = 0

Splunk version is 7.0.3

0 Karma

yannK
Splunk Employee
Splunk Employee

food for thoughts

  • Have you thought monitoring the rotated file too ?
    Splunk had a logic to avoid reindexing the file
    Sometimes it helps if the last events of the file were not indexed before the file rotated.

  • if the problem is during the rotation process, maybe disable the monitoring before the rotation and re-enable after.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...