Splunk Enterprise

Forcing LWF to resend (and Indexer to re-index) segment of corrupted data

TR_Splunker
Engager

We recently rebuilt several endpoints and cloned the configs on them. Unfortunately, the input.conf file had the same [default] host= for all 18 servers because all the files were cloned from one server. While the data is present, it is all being lumped under one hostname.

We've fixed the input.conf file and now all the data is being handled correctly, but we want to re-import about 2 weeks worth that was pulled in with the wrong hostname.

Is there a way to delete a range of data that is corrupted on the index servers, and force the forwarder to re-send it?

Tags (1)

jrodman
Splunk Employee
Splunk Employee

It's possible to hide some data so that has been already indexed with the delete command, this makes it no longer searchable.

http://www.splunk.com/base/Documentation/4.1.5/SearchReference/Delete http://www.splunk.com/base/Documentation/4.1.5/Admin/RemovedatafromSplunk

It isn't reversible (and off by default), so measure twice, cut once.

You can force reindexing by a couple of different methods. You could reindex everything using a splunk clean eventdata on your forwarders. You could force reindexing of specific files by copying them to $SPLUNK_HOME/var/log/splunk, though the paths will be a bit different. You can tell splunk to index a particular file regardless of the duplication logic with the oneshot input method: splunk help add oneshot

Lastly, a bit dirty, you could get somewhat sneaky and defeat splunk's redundancy checking. If you modify the first 256 bytes of your logfiles, eg by inserting a single character of whitespace at the start of them, it will reindex those files, assuming they are totally new.

Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...