I have noticed that some forwarders are not sending all of the log files. The log files are rotated hourly and I can see in the forwarder's log that it notices the log rotation and sends the file over. But once in a while it will not send it over and I can see that there is no corresponding event for that hour in the splunkd.log file on the forwarder stating that it has noticed a change in the log:
03-26-2012 15:25:25.044 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 15:49:11.554 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 16:00:00.449 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 16:00:00.449 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 17:00:00.754 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 17:00:00.754 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 17:45:30.230 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 18:00:00.148 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 18:00:00.148 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 19:10:42.208 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 19:10:42.208 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 21:00:55.261 +0000 INFO BatchReader - Removed from queue file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 21:00:55.262 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 21:00:55.262 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 22:02:56.527 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/ea/nova/nucleus/serv/nucleus.log'.
03-26-2012 22:02:56.527 +0000 INFO WatchedFile - Will begin reading at offset=0 for
file='/opt/ea/nova/nucleus/serv/nucleus.log'.
You can see that there are no events for 20:00 and I can see the missing gap in the timeline when I do a search.
Any thoughts? 😕
Thanks!
