Deployment Architecture

Splunk forwarder Skipping Log files occasionally


HI Experts
I am having an issue in indexing the log file which gets rotated ever hour. The log file error.log gets rotated every hour at top of the hour and a new file is created with the same name(error.log). The old file gets renamed and zipped to error.log._timestamp.gz.
Sometimes splunk does not index the file for an hour and resumes the indexing once the file is again rotated so the complete 1 hour logs gets skipped. Before splunk resumes the indexing following error message is logged.

WatchedFile - Checksum for seekptr didn't match, will re-read entire file

Every file has a different content because each event has a timestamp so first 256 characters should not much the fishbucket.

_TCP_ROUTING = indexers123
index = indexname
sourcetype = sourcetypename
initCrcLength = 1024
disabled = 0

Splunk version is 7.0.3

0 Karma

Splunk Employee
Splunk Employee

food for thoughts

  • Have you thought monitoring the rotated file too ?
    Splunk had a logic to avoid reindexing the file
    Sometimes it helps if the last events of the file were not indexed before the file rotated.

  • if the problem is during the rotation process, maybe disable the monitoring before the rotation and re-enable after.

0 Karma
Get Updates on the Splunk Community!

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...