Deployment Architecture

How do I delete data from just one specific index on one specific indexer in a cluster to free up disk space?

Path Finder

I have indexer cluster consisting of 3 indexers(ind-a, ind-b, ind-c). Replication and search factor is set to: 3. All works fine, no issues.

But, I keep running out of disk space on ind-c only. Following are the steps I was thinking of applying but wanted to make sure this is the right approach.

Is that the indexes.conf in ~/etc/slave-apps/_cluster/local will override the indexes.conf in etc/system/local, thus creating a loop and not resolving my issue. Can someone help with my flow

1. Select 2 large indexes that I can afford to lose since they are already replicated on ind-a & ind-B to free up disk space just on ind-c
2. Prevent any new data from being indexed to either index on ind-c

1. ssh ind-c
2. stop splunk
3. cd $SPLUNK_DB
4. cd big-index-dir-1 (80GB)
5. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
6. cd big-index-dir-2 (100GB)
7. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
8. du -sh $SPLUNK_DB (Plenty of space now)
9. Do not edit the ~/etc/slave-apps/_cluster/local/indexes.conf (this is controlled by the MN)
10. Instead do this: cd ~/etc/system/local
11. Create a new indexes.conf and insert:

# turned off 
repFactor  = 0           
homePath   = $SPLUNK_DB/big-index-dir-1/db
coldPath   = $SPLUNK_DB/big-index-dir-1/colddb
thawedPath = $SPLUNK_DB/big-index-dir-1/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-1/frozen-archivedb
maxTotalDataSizeMB = 80000
frozenTimePeriodInSecs = 259200

# turned OFF
repFactor  = 0
homePath   = $SPLUNK_DB/big-index-dir-2/db
coldPath   = $SPLUNK_DB/big-index-dir-2/colddb
thawedPath = $SPLUNK_DB/big-index-dir-2/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-2/frozen-archivedb
maxTotalDataSizeMB = 100000
frozenTimePeriodInSecs = 259200

splunk start

0 Karma
Get Updates on the Splunk Community!

Index This | A sphere has three, a circle has two, and a point has zero. What is it?

September 2023 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...