For some reason I don't have any indexed data from September 22 through October 25. The user who brought this to my attention swears that the data was there yesterday. No matter how I search, the data is not there. The logs are light forwarded to the indexer. How can I force the re-forwarding and or re-indexing of these log files?
You can reindex files in a few ways. I would be more concerned to make sure the data actually got deleted/removed before proceeding. There are answers topics on re-indexing the same files that you can search for. To resolve your problem:
1 - Search over all time AND your specified date range to make sure your data is really gone...use the source= and index=* parameters.
index=* source=/dir/my/file.log | timechart count by source
2 - Check metadata to see when Splunk last saw something:
| metadata type=sources index=* | convert ctime(lastTime) ctime(recentTime) ctime(firstTime)
Look for your source in the above output and verify when it last saw an event.
3 - If the above searches don't contain your source, then it probably got rolled out due to your retention policy. Check the $SPLUNK_HOME/var/log/splunkd.log file to see when the last bucket was rolled out of warm or cold.
This is in the log. How do I get my data back?
10-25-2010 10:10:08.452 INFO databasePartitionPolicy - Moving db with id of 43: /opt/splunk/var/lib/splunk/internaldb/db/hotv1_43 to warm: size exceeded: maxDataSize=104857600 bytes, bucketSize=106525084 bytes
10-25-2010 10:10:08.452 WARN databasePartitionPolicy - About to move db at /opt/splunk/var/lib/splunk/internaldb/db/hotv1_43 to warm