Splunk Enterprise

Forcing LWF to resend (and Indexer to re-index) segment of corrupted data

TR_Splunker
Engager

We recently rebuilt several endpoints and cloned the configs on them. Unfortunately, the input.conf file had the same [default] host= for all 18 servers because all the files were cloned from one server. While the data is present, it is all being lumped under one hostname.

We've fixed the input.conf file and now all the data is being handled correctly, but we want to re-import about 2 weeks worth that was pulled in with the wrong hostname.

Is there a way to delete a range of data that is corrupted on the index servers, and force the forwarder to re-send it?

Tags (1)

jrodman
Splunk Employee
Splunk Employee

It's possible to hide some data so that has been already indexed with the delete command, this makes it no longer searchable.

http://www.splunk.com/base/Documentation/4.1.5/SearchReference/Delete http://www.splunk.com/base/Documentation/4.1.5/Admin/RemovedatafromSplunk

It isn't reversible (and off by default), so measure twice, cut once.

You can force reindexing by a couple of different methods. You could reindex everything using a splunk clean eventdata on your forwarders. You could force reindexing of specific files by copying them to $SPLUNK_HOME/var/log/splunk, though the paths will be a bit different. You can tell splunk to index a particular file regardless of the duplication logic with the oneshot input method: splunk help add oneshot

Lastly, a bit dirty, you could get somewhat sneaky and defeat splunk's redundancy checking. If you modify the first 256 bytes of your logfiles, eg by inserting a single character of whitespace at the start of them, it will reindex those files, assuming they are totally new.

Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...