Splunk Search

Find and Parse Missing Events

trilogy
New Member

Splunk was shut down for a few weeks on my server, and now I am missing events from my log files for the time it was not running. How can I tell Splunk to go find and parse all of the missing events?

Tags (3)
0 Karma

dbryan
Path Finder

How is the data getting into Splunk? If it was coming via a forwarder, the forwarder should have automatically noticed when the indexer was down and held the data until it was back up.

If you have a single Splunk instance handling the input handling from start to finish, you may want to check out the followTail setting for its stanza in inputs.conf:

http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf

It may not work in your use case, but in situations like this I often find it's easier to just clear the index, as well as the "_thefishbucket" index which is used to keep track of data that has been indexed:

splunk stop
splunk clean eventdata -index main #or whatever your index is
splunk clean eventdata -index _thefishbucket
splunk start

Beware, though, this will cause all of your inputs to be re-indexed, and remove all data from the the main index, or whichever other index you specify.

Alternatively, if you're missing data from some entire log files, you could use the CLI:

http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorfilesanddirectoriesusingtheCLI

splunk add oneshot can be used to add a single file. Make sure you specify the index, host and sourcetype if necessary.

0 Karma

dbryan
Path Finder

Yes, you can use the splunk binary on the forwarder for this. Make sure you specify the sourcetype, index, and anything else that would normally be set in inputs.conf.

0 Karma

trilogy
New Member

I am using a forwarder. Can I use add oneshot to add a file from another server?

0 Karma

Drainy
Champion

"the forwarder should have automatically noticed when the indexer was down and held the data until it was back up." - If Splunk was down for a few weeks then the local queue would have filled up and the forwarder would have begun to drop events, leading to the missing data.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...