Find and Parse Missing Events

[trilogy]

Splunk was shut down for a few weeks on my server, and now I am missing events from my log files for the time it was not running. How can I tell Splunk to go find and parse all of the missing events?

[dbryan]

How is the data getting into Splunk? If it was coming via a forwarder, the forwarder should have automatically noticed when the indexer was down and held the data until it was back up.

If you have a single Splunk instance handling the input handling from start to finish, you may want to check out the followTail setting for its stanza in inputs.conf:

http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf

It may not work in your use case, but in situations like this I often find it's easier to just clear the index, as well as the "_thefishbucket" index which is used to keep track of data that has been indexed:

splunk stop
splunk clean eventdata -index main #or whatever your index is
splunk clean eventdata -index _thefishbucket
splunk start

Beware, though, this will cause all of your inputs to be re-indexed, and remove all data from the the main index, or whichever other index you specify.

Alternatively, if you're missing data from some entire log files, you could use the CLI:

http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorfilesanddirectoriesusingtheCLI

splunk add oneshot can be used to add a single file. Make sure you specify the index, host and sourcetype if necessary.

[dbryan]

Yes, you can use the splunk binary on the forwarder for this. Make sure you specify the sourcetype, index, and anything else that would normally be set in inputs.conf.

[trilogy]

I am using a forwarder. Can I use add oneshot to add a file from another server?

[Drainy]

"the forwarder should have automatically noticed when the indexer was down and held the data until it was back up." - If Splunk was down for a few weeks then the local queue would have filled up and the forwarder would have begun to drop events, leading to the missing data.