I'm trying to create a dashboard for missing universal Forwarders.
I tried using _internal logs, but for a few of the hosts I'm not getting internal logs regularly.
I'm getting the below entries often on those hosts. What does it mean. What do I need to do?
Could anyone please explain?
10-09-2014 08:30:20.317 +0100 INFO WatchedFile - Will begin reading at offset=24997255 for file='/opt/splunkforwarder/var/log/splunk/metrics.log.1'.
host=abc source=/opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
10-09-2014 08:30:19.781 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
host = abc source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
10-09-2014 08:30:19.756 +0100 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
host = abc source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd
This is might unusual. The system's own logfiles are having seekpointer mismatches.
This suggests something unpleasant, like two splunks fighting over one set of logfile dirs, sysadmins manually modifying the logfiles, a choice to manually modify (eg rotate) splunk logfiles outside of splunk, or problems in the filesystem the data is stored on.