Deployment Architecture

Splunk forwarder Skipping Log files occasionally

vaibhavagg2006
Communicator

HI Experts
I am having an issue in indexing the log file which gets rotated ever hour. The log file error.log gets rotated every hour at top of the hour and a new file is created with the same name(error.log). The old file gets renamed and zipped to error.log._timestamp.gz.
Sometimes splunk does not index the file for an hour and resumes the indexing once the file is again rotated so the complete 1 hour logs gets skipped. Before splunk resumes the indexing following error message is logged.

WatchedFile - Checksum for seekptr didn't match, will re-read entire file

Every file has a different content because each event has a timestamp so first 256 characters should not much the fishbucket.

[monitor:///data/logs/]
_TCP_ROUTING = indexers123
index = indexname
sourcetype = sourcetypename
initCrcLength = 1024
whitelist=\.log$
disabled = 0

Splunk version is 7.0.3

0 Karma

yannK
Splunk Employee
Splunk Employee

food for thoughts

  • Have you thought monitoring the rotated file too ?
    Splunk had a logic to avoid reindexing the file
    Sometimes it helps if the last events of the file were not indexed before the file rotated.

  • if the problem is during the rotation process, maybe disable the monitoring before the rotation and re-enable after.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...