Deployment Architecture
Highlighted

How do I clean a clustered index?

Splunk Employee
Splunk Employee

What's the best way to completely clean an index in a clustered environment?

Tags (3)
Highlighted

Re: How do I clean a clustered index?

Ultra Champion

Would cleaning it on each of the nodes not propagate to the replicas?

Or would it be better/possible to set replication/search factor to 1, and then (after a little while, perhaps) clean the index on the nodes?

Highlighted

Re: How do I clean a clustered index?

Splunk Employee
Splunk Employee

Currently, there is not really a good way to do this as cleaning the event data from an index will just get replicated back from another cluster node.

That being said, there are two not quite so nice ways of doing it.

  1. With a user that has the can_delete permissions, pipe all the event data to be removed to the delete command. Naturally, this means that all the caveats for using the delete command apply. (Data is not removed from disk, etc.)
  2. Make sure you stop indexing data to the index your are about to clean and alter your data retention policy to be extremely short. This will roll all the buckets to frozen and hence clear out the index. Once all the data has been removed from the index on all the peers, the retention policy can be set back to its original settings in order to allow for new data to be indexed.

For the 1st option, you may find the following link useful:
http://docs.splunk.com/Documentation/Splunk/5.0.3/Indexer/RemovedatafromSplunk#Howtodelete
For the 2nd options, take a look at the documentation here:
http://docs.splunk.com/Documentation/Splunk/5.0.3/Indexer/Setaretirementandarchivingpolicy

View solution in original post

Highlighted

Re: How do I clean a clustered index?

Splunk Employee
Splunk Employee

By design, it is meant to be difficult to delete data in a cluster. The point of clustering is to make data resistant to loss by copying and replicating it. So besides these suggestions, you would have to stop all indexers and clean the indexes on each one manually to remove all the replicas to prevent recovery.

0 Karma
Highlighted

Re: How do I clean a clustered index?

Splunk Employee
Splunk Employee

The only problem with cleaning each indexer node with the entire cluster down is that the cluster master may not have any knowledge of the data being unavailable.

0 Karma
Highlighted

Re: How do I clean a clustered index?

Splunk Employee
Splunk Employee

The cluster master does not keep the knowledge of the data location without the indexer nodes. If they are all down, then all that the CM knows is that there is no data available anywhere. When the indexers recover, they tell the master that none of them have any data, and that is all the CM knows. The CM's job is not to track each piece of data, but to ensure that any piece of data that is reported to exist is sufficiently replicated, and to tell the search head where they are. It gets the knowledge to do this from the indexers.

0 Karma
Highlighted

Re: How do I clean a clustered index?

Explorer

Option 2 worked for me, and in my case a rolling restart wasn't even initiated.

0 Karma
Highlighted

Re: How do I clean a clustered index?

Splunk Employee
Splunk Employee

I downvoted this post because do not use | delete to clean an index.

0 Karma
Highlighted

Re: How do I clean a clustered index?

Engager

What would happen if I run on each indexer at nearly same time Splunk offline command,
and than run ./splunk clean eventdata -index command on each indexer?

When data is removed I would start every indexer.
This actions would take less than 10 minutes, so the master node would not detect indexer failure.

0 Karma
Highlighted

Re: How do I clean a clustered index?

Splunk Employee
Splunk Employee

@manjosk8

The problem you will run in to is that the cluster master may retain the info on data availability of the peers. In turn this will cause the search head to look for data where it no longer exists.

The bigger issue is that the actions taking 10min or less would be insufficient. On start the buckets are checked and replication begins. Also, the default peer heartbeat is 30 or 60 seconds depending on the version of Splunk you are running.

In the end, these actions would either create a faulty cluster setup or it would not delete the data as expected.

0 Karma