Getting Data In

How to delete data from an index within a index cluster using SmartStore?

Jamie
Explorer

Hello.  I am running 8.2.2 on Linux.  We have four clustered indexers and are using SmartStore.  I would like to empty an index (and recover the disk space).  I have thus chosen to remove the old_data index from the cluster, then add it back again.  I have performed these steps:

1. Stop any data being sent to the index.
2. Edit indexes.conf and delete the index's stanza (via the CM) then apply the changes to the peer nodes (each restarts).
3. Remove the index's directories from each peer node.
4. Check on the SHC for events in the index (index=old_data); no events are returned (all time).
5. Once the cluster shows that all indexes are 'green', re-add the index as normnal (editing indexes.conf again and applying the update).

However, now searching the index on the SHC returns some/most of the events.  My guess is that the cache manager / the S3 storage also needs to be purged.   If so, how is this best achieved?

I have avoided using index=old_data | delete because I understand this will only mask the data from searches (and I want the disk space back too).

Many thanks for your time.

Labels (3)
0 Karma
1 Solution

gcusello
Esteemed Legend

Hi @Jamie,

put = 0 the retention for the index the you want to clean, setting  on the Master Node in the related stanza of indexes.conf:

FrozenTimePeriodInSecs = 0

then save and push the configurations to the indexers.

after few minutes, when the index is cleaned, you can set again the retention to the correct value.

Ciao.

Giuseppe

View solution in original post

gcusello
Esteemed Legend

Hi @Jamie,

put = 0 the retention for the index the you want to clean, setting  on the Master Node in the related stanza of indexes.conf:

FrozenTimePeriodInSecs = 0

then save and push the configurations to the indexers.

after few minutes, when the index is cleaned, you can set again the retention to the correct value.

Ciao.

Giuseppe

Jamie
Explorer

Ciao @gcusello,

Thank you for getting back to me.

Success!

Initially this did not work; the events continued to be returned from a search (I did wait 30 mins).  However, I had taken a tarball of the old_data directory on each indexer (plus old_data.dat) before starting.  So I:

- once again removed the old_data index from the cluster (i.e. updated indexes.conf from the CM).

- restored the tarball on each indexer.

- re-added the index back to indexes.conf.

- searched the data and saw the events as normal.

- edited indexes.conf setting FrozenTimePeriodInSecs = 0 for the old_data index.

However, I still saw the data with a search (but perhaps I should have waited longer, I beleive I waited 10+ minutes).  So I then changed FrozenTimePeriodInSecs = 1.  Perhaps a coincidence, but finally, the search returned no events.


Grazie!

Jamie.

0 Karma

gcusello
Esteemed Legend

Hi @Jamie,

good for you, see next time!

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...