Getting Data In

Forwarder missing log rotation

romantercero
Path Finder

I have noticed that some forwarders are not sending all of the log files. The log files are rotated hourly and I can see in the forwarder's log that it notices the log rotation and sends the file over. But once in a while it will not send it over and I can see that there is no corresponding event for that hour in the splunkd.log file on the forwarder stating that it has noticed a change in the log:

03-26-2012 15:25:25.044 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 15:49:11.554 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 16:00:00.449 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 16:00:00.449 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 17:00:00.754 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 17:00:00.754 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 17:45:30.230 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 18:00:00.148 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 18:00:00.148 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 19:10:42.208 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 19:10:42.208 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 21:00:55.261 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 21:00:55.262 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 21:00:55.262 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 22:02:56.527 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 22:02:56.527 +0000 INFO WatchedFile - Will begin reading at offset=0 for
file='/opt/ea/nova/nucleus/serv/nucleus.log'.

You can see that there are no events for 20:00 and I can see the missing gap in the timeline when I do a search.

Any thoughts? 😕
Thanks!

Tags (2)
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi romantercero

there is a known bug if log file are being rotated with 'logadm -c', see

"Monitor on files stops indexing files if the file is truncated while calculating the CRC. (SPL-44773)"

It is fixed in 4.3.3

cheers,

MuS

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...