Hello the data we're trying to index is just a single log file that grows continuously. It appears data is being transferred to Splunk, but at some point in the past the data just stopped showing up when I tried to view the data in the Search App. I'm not sure how to resolve this issue. I know we are not low on disk space, nor are we anywhere near the daily indexing limit. Help?
inputs.conf on windows server
[monitor://D:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log]
disabled = false
Snippet from metrics.log of server data in question (splunk server)
09-01-2011 08:38:58.788 -0400 INFO Metrics - group=tcpin_connections, some-ip:2927:9997, connectionType=cooked, sourcePort=2927, sourceHost=some-ip, sourceIp=some-ip, destPort=9997, _tcp_Bps=12.68, _tcp_KBps=0.01, _tcp_avg_thruput=0.24, kb=0.38, _tcp_Kprocessed=206874.00, _tcp_eps=0.03, build=105575, version=4.2.3, os=Windows, arch=x64, hostname=lebhq-notes, guid=some-guid, fwdType=uf, ssl=false, lastIndexer=some-other-ip:9997, ack=false
End of splunkd.log from my windows server with universalforwarder
08-22-2011 13:58:05.030 -0400 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
08-22-2011 13:58:05.545 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text="WARNING: The public key for Mary McCreery/LEB/CSGROUP found in directory names.nsf on server LEBHQ_M..."
08-22-2011 13:58:05.577 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text=" ..."
08-22-2011 13:58:05.577 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text="If the problem persists please notify your Notes Administrator of the following error: ..."
08-22-2011 13:58:05.577 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text="Routine: Export_Processing - Initialize_ExportProcessingScript ..."
08-22-2011 13:58:05.577 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text="Error number: 13 ..."
08-22-2011 13:58:05.577 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text="Line number: 25 ..."
08-22-2011 13:58:05.577 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text="Description: Type mismatch..."
08-22-2011 13:58:05.592 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text="WARNING: The public key for **redacted** found in directory names.nsf on server **redacted**..."
08-22-2011 13:58:05.608 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text=" ..."
08-22-2011 13:58:05.608 -0400 WARN DateParserVerbose - Failed to parse timestamp for event. Text="If the problem persists please notify your Notes Administrator of the following error: ..."
08-22-2011 13:58:09.545 -0400 INFO TcpOutputProc - Connected to idx=hostname:9997
08-22-2011 14:11:40.129 -0400 INFO BatchReader - Removed from queue file='D:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log'.
08-24-2011 05:04:43.025 -0400 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-24-2011 05:04:43.025 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-24-2011 05:04:43.040 -0400 INFO WatchedFile - Will begin reading at offset=24995178 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-25-2011 20:09:48.867 -0400 INFO WatchedFile - Will begin reading at offset=24996842 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-25-2011 20:09:49.226 -0400 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-25-2011 20:09:49.226 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-27-2011 11:19:02.751 -0400 INFO WatchedFile - Will begin reading at offset=24997689 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-27-2011 11:19:02.985 -0400 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-27-2011 11:19:02.985 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-29-2011 02:27:46.349 -0400 INFO WatchedFile - Will begin reading at offset=24995368 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-29-2011 02:27:46.740 -0400 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-29-2011 02:27:46.740 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-30-2011 17:40:36.268 -0400 WARN Logger - Error unlinking "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.5": The system cannot find the file specified.
08-30-2011 17:40:37.518 -0400 INFO WatchedFile - Will begin reading at offset=24995300 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-30-2011 17:40:38.002 -0400 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-30-2011 17:40:38.002 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
09-01-2011 08:44:09.931 -0400 INFO WatchedFile - Will begin reading at offset=24996376 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
... View more