Developing for Splunk Enterprise

Official way to clean indexed data from index cluster

napomokoetle
Communicator

Hi,

I have an indexer cluster with two indexers, a separate Cluster Master Node and one separate Search Head.

Both Indexers have 500 GB space and the disk space is now fully used and we're now dropping events.

Can I delete data from the _internal index on the clustered indexers to free some space? If so how do I safely do so?
Can I delete data from some non-interal indexers on the clustered indexers to free some space? If so how do I execute the cleaning with minimal risk of breaking the environment?

I have used the "./splunk clean eventdata -index myIndex -f" command on STANDALONE environments without a problem, but I am not sure if that would be SAFE on a clustered indexer environment due to replication/synchronization/etc.

Look forward to hear from you. Thanks advance for your help.

Kind regards,
J. Napo Mokoetle

0 Karma
1 Solution

gfuente
Motivator

Hello

The best way (i found) to do this, is to set a smaller retention policy in the indexes.conf (through the master apps) to the indexes you want to delete events, apply the changes, wait some minutes until the indexers delete the events, and afterwards, change back the retention policy to the final value

You need to include this setting in your indexes.conf

frozenTimePeriodInSecs =

For example, you could set it to 86400, to delete everything older than 1 day

regards

View solution in original post

gfuente
Motivator

Hello

The best way (i found) to do this, is to set a smaller retention policy in the indexes.conf (through the master apps) to the indexes you want to delete events, apply the changes, wait some minutes until the indexers delete the events, and afterwards, change back the retention policy to the final value

You need to include this setting in your indexes.conf

frozenTimePeriodInSecs =

For example, you could set it to 86400, to delete everything older than 1 day

regards

View solution in original post

napomokoetle
Communicator

Thanks for your swift response gfuente.

Just to make sure I'm with you.

The indexes.conf to edit on the master is the one in the path

/opt/splunk/etc/master-apps/_cluster/local/indexes.conf

right?

0 Karma

gfuente
Motivator

Yeah, could be that, or any other in any app under master apps, :

/opt/splunk/etc/master-apps//local/indexes.conf

Basically, wherever you already are configuring your existing indexes

Regards

0 Karma

napomokoetle
Communicator

Hi gfuente,

Your method worked like a charm. Thanks a great deal!

gfuente
Motivator

Good to know it

0 Karma

napomokoetle
Communicator

I am running Splunk 6.3.2 on Linux Centos.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!