Hole in my data

[kmattern]

For some reason I don't have any indexed data from September 22 through October 25. The user who brought this to my attention swears that the data was there yesterday. No matter how I search, the data is not there. The logs are light forwarded to the indexer. How can I force the re-forwarding and or re-indexing of these log files?

[Simeon]

You can reindex files in a few ways. I would be more concerned to make sure the data actually got deleted/removed before proceeding. There are answers topics on re-indexing the same files that you can search for. To resolve your problem:

1 - Search over all time AND your specified date range to make sure your data is really gone...use the source= and index=* parameters.

index=* source=/dir/my/file.log | timechart count by source

2 - Check metadata to see when Splunk last saw something:

| metadata type=sources index=* | convert ctime(lastTime) ctime(recentTime) ctime(firstTime)

Look for your source in the above output and verify when it last saw an event.

3 - If the above searches don't contain your source, then it probably got rolled out due to your retention policy. Check the $SPLUNK_HOME/var/log/splunkd.log file to see when the last bucket was rolled out of warm or cold.

[kmattern]

This is in the log. How do I get my data back?

10-25-2010 10:10:08.452 INFO databasePartitionPolicy - Moving db with id of 43: /opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_43 to warm: size exceeded: maxDataSize=104857600 bytes, bucketSize=106525084 bytes

10-25-2010 10:10:08.452 WARN databasePartitionPolicy - About to move db at /opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_43 to warm