Splunk Enterprise

Forcing LWF to resend (and Indexer to re-index) segment of corrupted data


We recently rebuilt several endpoints and cloned the configs on them. Unfortunately, the input.conf file had the same [default] host= for all 18 servers because all the files were cloned from one server. While the data is present, it is all being lumped under one hostname.

We've fixed the input.conf file and now all the data is being handled correctly, but we want to re-import about 2 weeks worth that was pulled in with the wrong hostname.

Is there a way to delete a range of data that is corrupted on the index servers, and force the forwarder to re-send it?

Tags (1)

Splunk Employee
Splunk Employee

It's possible to hide some data so that has been already indexed with the delete command, this makes it no longer searchable.

http://www.splunk.com/base/Documentation/4.1.5/SearchReference/Delete http://www.splunk.com/base/Documentation/4.1.5/Admin/RemovedatafromSplunk

It isn't reversible (and off by default), so measure twice, cut once.

You can force reindexing by a couple of different methods. You could reindex everything using a splunk clean eventdata on your forwarders. You could force reindexing of specific files by copying them to $SPLUNK_HOME/var/log/splunk, though the paths will be a bit different. You can tell splunk to index a particular file regardless of the duplication logic with the oneshot input method: splunk help add oneshot

Lastly, a bit dirty, you could get somewhat sneaky and defeat splunk's redundancy checking. If you modify the first 256 bytes of your logfiles, eg by inserting a single character of whitespace at the start of them, it will reindex those files, assuming they are totally new.

Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...