Official way to clean indexed data from index cluster

[napomokoetle]

Hi,

I have an indexer cluster with two indexers, a separate Cluster Master Node and one separate Search Head.

Both Indexers have 500 GB space and the disk space is now fully used and we're now dropping events.

Can I delete data from the _internal index on the clustered indexers to free some space? If so how do I safely do so?
Can I delete data from some non-interal indexers on the clustered indexers to free some space? If so how do I execute the cleaning with minimal risk of breaking the environment?

I have used the "./splunk clean eventdata -index myIndex -f" command on STANDALONE environments without a problem, but I am not sure if that would be SAFE on a clustered indexer environment due to replication/synchronization/etc.

Look forward to hear from you. Thanks advance for your help.

Kind regards,
J. Napo Mokoetle

[gfuente · Accepted solution]

Hello

The best way (i found) to do this, is to set a smaller retention policy in the indexes.conf (through the master apps) to the indexes you want to delete events, apply the changes, wait some minutes until the indexers delete the events, and afterwards, change back the retention policy to the final value

You need to include this setting in your indexes.conf

frozenTimePeriodInSecs =

For example, you could set it to 86400, to delete everything older than 1 day

regards

[napomokoetle]

Thanks for your swift response gfuente.

Just to make sure I'm with you.

The indexes.conf to edit on the master is the one in the path

/opt/splunk/etc/master-apps/_cluster/local/indexes.conf

right?

[gfuente]

Yeah, could be that, or any other in any app under master apps, :

/opt/splunk/etc/master-apps//local/indexes.conf

Basically, wherever you already are configuring your existing indexes

Regards

[napomokoetle]

Hi gfuente,

Your method worked like a charm. Thanks a great deal!

[gfuente]

Good to know it

[napomokoetle]

I am running Splunk 6.3.2 on Linux Centos.