Deployment Architecture

Splunk forwarder Skipping Log files occasionally

vaibhavagg2006
Communicator

HI Experts
I am having an issue in indexing the log file which gets rotated ever hour. The log file error.log gets rotated every hour at top of the hour and a new file is created with the same name(error.log). The old file gets renamed and zipped to error.log._timestamp.gz.
Sometimes splunk does not index the file for an hour and resumes the indexing once the file is again rotated so the complete 1 hour logs gets skipped. Before splunk resumes the indexing following error message is logged.

WatchedFile - Checksum for seekptr didn't match, will re-read entire file

Every file has a different content because each event has a timestamp so first 256 characters should not much the fishbucket.

[monitor:///data/logs/]
_TCP_ROUTING = indexers123
index = indexname
sourcetype = sourcetypename
initCrcLength = 1024
whitelist=\.log$
disabled = 0

Splunk version is 7.0.3

0 Karma

yannK
Splunk Employee
Splunk Employee

food for thoughts

  • Have you thought monitoring the rotated file too ?
    Splunk had a logic to avoid reindexing the file
    Sometimes it helps if the last events of the file were not indexed before the file rotated.

  • if the problem is during the rotation process, maybe disable the monitoring before the rotation and re-enable after.

0 Karma
Get Updates on the Splunk Community!

Splunk MCP & Agentic AI: Machine Data Without Limits

  Discover how the Splunk Model Context Protocol (MCP) Server can revolutionize the way your organization ...

Finding Based Detections General Availability

Overview  We’ve come a long way, folks, but here in Enterprise Security 8.4 I’m happy to announce Finding ...

Get Your Hands Dirty (and Your Shoes Comfy): The Splunk Experience

Hands-On Learning and Technical Seminars  Sometimes, you just need to see the code. For those looking for a ...