Getting Data In

Forwarder missing log rotation

romantercero
Path Finder

I have noticed that some forwarders are not sending all of the log files. The log files are rotated hourly and I can see in the forwarder's log that it notices the log rotation and sends the file over. But once in a while it will not send it over and I can see that there is no corresponding event for that hour in the splunkd.log file on the forwarder stating that it has noticed a change in the log:

03-26-2012 15:25:25.044 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 15:49:11.554 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 16:00:00.449 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 16:00:00.449 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 17:00:00.754 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 17:00:00.754 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 17:45:30.230 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 18:00:00.148 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 18:00:00.148 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 19:10:42.208 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 19:10:42.208 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 21:00:55.261 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 21:00:55.262 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 21:00:55.262 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 22:02:56.527 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 22:02:56.527 +0000 INFO WatchedFile - Will begin reading at offset=0 for
file='/opt/ea/nova/nucleus/serv/nucleus.log'.

You can see that there are no events for 20:00 and I can see the missing gap in the timeline when I do a search.

Any thoughts? 😕
Thanks!

Tags (2)
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi romantercero

there is a known bug if log file are being rotated with 'logadm -c', see

"Monitor on files stops indexing files if the file is truncated while calculating the CRC. (SPL-44773)"

It is fixed in 4.3.3

cheers,

MuS

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...