Forwarder missing log rotation

romantercero — Wed, 28 Mar 2012 22:20:56 GMT

I have noticed that some forwarders are not sending all of the log files. The log files are rotated hourly and I can see in the forwarder's log that it notices the log rotation and sends the file over. But once in a while it will not send it over and I can see that there is no corresponding event for that hour in the splunkd.log file on the forwarder stating that it has noticed a change in the log:

03-26-2012 15:25:25.044 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 15:49:11.554 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 16:00:00.449 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 16:00:00.449 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 17:00:00.754 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 17:00:00.754 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 17:45:30.230 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 18:00:00.148 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 18:00:00.148 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 19:10:42.208 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 19:10:42.208 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 21:00:55.261 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 21:00:55.262 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 21:00:55.262 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 22:02:56.527 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.

03-26-2012 22:02:56.527 +0000 INFO WatchedFile - Will begin reading at offset=0 for
file='/opt/ea/nova/nucleus/serv/nucleus.log'.

You can see that there are no events for 20:00 and I can see the missing gap in the timeline when I do a search.

Any thoughts? 😕
Thanks!

Re: Forwarder missing log rotation

MuS — Fri, 07 Sep 2012 12:47:01 GMT

Hi romantercero

there is a known bug if log file are being rotated with 'logadm -c', see

"Monitor on files stops indexing files if the file is truncated while calculating the CRC. (SPL-44773)"

It is fixed in 4.3.3

cheers,

MuS

topic Re: Forwarder missing log rotation in Getting Data In

Forwarder missing log rotation

Re: Forwarder missing log rotation