How to delete data from an index within a index cluster using SmartStore?

[Jamie]

Hello.  I am running 8.2.2 on Linux.  We have four clustered indexers and are using SmartStore.  I would like to empty an index (and recover the disk space).  I have thus chosen to remove the old_data index from the cluster, then add it back again.  I have performed these steps:

1. Stop any data being sent to the index.
2. Edit indexes.conf and delete the index's stanza (via the CM) then apply the changes to the peer nodes (each restarts).
3. Remove the index's directories from each peer node.
4. Check on the SHC for events in the index (index=old_data); no events are returned (all time).
5. Once the cluster shows that all indexes are 'green', re-add the index as normnal (editing indexes.conf again and applying the update).

However, now searching the index on the SHC returns some/most of the events.  My guess is that the cache manager / the S3 storage also needs to be purged.   If so, how is this best achieved?

I have avoided using index=old_data | delete because I understand this will only mask the data from searches (and I want the disk space back too).

Many thanks for your time.

[gcusello · Accepted solution]

Hi @Jamie,

put = 0 the retention for the index the you want to clean, setting  on the Master Node in the related stanza of indexes.conf:

FrozenTimePeriodInSecs = 0

then save and push the configurations to the indexers.

after few minutes, when the index is cleaned, you can set again the retention to the correct value.

Ciao.

Giuseppe

[Jamie]

Ciao @gcusello,

Thank you for getting back to me.

Success!

Initially this did not work; the events continued to be returned from a search (I did wait 30 mins).  However, I had taken a tarball of the old_data directory on each indexer (plus old_data.dat) before starting.  So I:

- once again removed the old_data index from the cluster (i.e. updated indexes.conf from the CM).

- restored the tarball on each indexer.

- re-added the index back to indexes.conf.

- searched the data and saw the events as normal.

- edited indexes.conf setting FrozenTimePeriodInSecs = 0 for the old_data index.

However, I still saw the data with a search (but perhaps I should have waited longer, I beleive I waited 10+ minutes).  So I then changed FrozenTimePeriodInSecs = 1.  Perhaps a coincidence, but finally, the search returned no events.

Grazie!

Jamie.

[gcusello]

Hi @Jamie,

good for you, see next time!

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉