Data not showing as being indexed, or showing up in search app.

[procha]

Hello the data we're trying to index is just a single log file that grows continuously. It appears data is being transferred to Splunk, but at some point in the past the data just stopped showing up when I tried to view the data in the Search App. I'm not sure how to resolve this issue. I know we are not low on disk space, nor are we anywhere near the daily indexing limit. Help?

inputs.conf on windows server

[monitor://D:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log]
disabled = false

Snippet from metrics.log of server data in question (splunk server)

09-01-2011 08:38:58.788 -0400 INFO  Metrics - group=tcpin_connections, some-ip:2927:9997, connectionType=cooked, sourcePort=2927, sourceHost=some-ip, sourceIp=some-ip, destPort=9997, _tcp_Bps=12.68, _tcp_KBps=0.01, _tcp_avg_thruput=0.24, kb=0.38, _tcp_Kprocessed=206874.00, _tcp_eps=0.03, build=105575, version=4.2.3, os=Windows, arch=x64, hostname=lebhq-notes, guid=some-guid, fwdType=uf, ssl=false, lastIndexer=some-other-ip:9997, ack=false

End of splunkd.log from my windows server with universalforwarder

08-22-2011 13:58:05.030 -0400 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
08-22-2011 13:58:05.545 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text="WARNING: The public key for Mary McCreery/LEB/CSGROUP found in directory names.nsf on server LEBHQ_M..."
08-22-2011 13:58:05.577 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text=" ..."
08-22-2011 13:58:05.577 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text="If the problem persists please notify your Notes Administrator of the following error: ..."
08-22-2011 13:58:05.577 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text="Routine: Export_Processing - Initialize_ExportProcessingScript ..."
08-22-2011 13:58:05.577 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text="Error number: 13 ..."
08-22-2011 13:58:05.577 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text="Line number: 25 ..."
08-22-2011 13:58:05.577 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text="Description: Type mismatch..."
08-22-2011 13:58:05.592 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text="WARNING: The public key for **redacted** found in directory names.nsf on server **redacted**..."
08-22-2011 13:58:05.608 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text=" ..."
08-22-2011 13:58:05.608 -0400 WARN  DateParserVerbose - Failed to parse timestamp for event.  Text="If the problem persists please notify your Notes Administrator of the following error: ..."
08-22-2011 13:58:09.545 -0400 INFO  TcpOutputProc - Connected to idx=hostname:9997
08-22-2011 14:11:40.129 -0400 INFO  BatchReader - Removed from queue file='D:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT\console.log'.
08-24-2011 05:04:43.025 -0400 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-24-2011 05:04:43.025 -0400 INFO  WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-24-2011 05:04:43.040 -0400 INFO  WatchedFile - Will begin reading at offset=24995178 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-25-2011 20:09:48.867 -0400 INFO  WatchedFile - Will begin reading at offset=24996842 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-25-2011 20:09:49.226 -0400 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-25-2011 20:09:49.226 -0400 INFO  WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-27-2011 11:19:02.751 -0400 INFO  WatchedFile - Will begin reading at offset=24997689 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-27-2011 11:19:02.985 -0400 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-27-2011 11:19:02.985 -0400 INFO  WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-29-2011 02:27:46.349 -0400 INFO  WatchedFile - Will begin reading at offset=24995368 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-29-2011 02:27:46.740 -0400 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-29-2011 02:27:46.740 -0400 INFO  WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-30-2011 17:40:36.268 -0400 WARN  Logger - Error unlinking "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.5": The system cannot find the file specified.
08-30-2011 17:40:37.518 -0400 INFO  WatchedFile - Will begin reading at offset=24995300 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.
08-30-2011 17:40:38.002 -0400 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
08-30-2011 17:40:38.002 -0400 INFO  WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
09-01-2011 08:44:09.931 -0400 INFO  WatchedFile - Will begin reading at offset=24996376 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.

[araitz]

In the future, take care to remove sample IPs and names from your sample data 🙂

[lguinn2]

Have you tried searching using "All time"? I wonder if the timestamps are being improperly parsed. This would mean that the data is being indexed, but with the wrong dates. Try several broad searches, and see if you can turn up the data.

You may need to set some of the data extraction parameters for this input. See How Timestamp Extraction Works for more details.

Also, enable the Splunk deployment monitor app. It can give you some good information about how much data is flowing from your forwarders to your indexers. One of the charts it shows is how much data is being forwarded by sourcetype. This might be very helpful.

[procha]

Thanks so much for your reply. I did check out the "all time" search, and it is definitely not showing data being indexed at all. I also looked at the deployment monitor, and confirmed that the splunk server isn't indexing the data. Would you suggest that I redo the universalforwarder install and use the "follow tail" command when I first set it up?

[RicoSuave]

Try adding the following attributes to your stanza

followTail=1
alwaysOpenFile=1

This is straight from the docs:
followTail = [0|1]
* Determines whether to start monitoring at the beginning of a file or at the end (and then index all events
that come in after that).
* If set to 1, monitoring begins at the end of the file (like tail -f).
* If set to 0, Splunk will always start at the beginning of the file.
* This only applies to files the first time Splunk sees them. After that, Splunk's internal file position
records keep track of the file.
* Defaults to 0.

alwaysOpenFile = [0|1]
* Opens a file to check whether it has already been indexed.
* Only useful for files that don't update modtime.
* Only needed when monitoring files on Windows, mostly for IIS logs.
* This flag should only be used as a last resort, as it increases load and slows down indexing.
* Defaults to 0.

[lguinn2]

I don't think that either of these settings will help in this case. followTail only applies the first time that Splunk sees a file; it is ignored thereafter.

[procha]

Hello, thanks for the reply. I added that to my inputs.conf on my windows server with the universalforwarder. I then restarted both splunk instances (universalforwarder and my indexing server), but I am still not seeing any new data in the search app.