I have indexer cluster consisting of 3 indexers(ind-a, ind-b, ind-c). Replication and search factor is set to: 3. All works fine, no issues.
But, I keep running out of disk space on ind-c only. Following are the steps I was thinking of applying but wanted to make sure this is the right approach.
Is that the indexes.conf in ~/etc/slave-apps/_cluster/local will override the indexes.conf in etc/system/local, thus creating a loop and not resolving my issue. Can someone help with my flow
1. Select 2 large indexes that I can afford to lose since they are already replicated on ind-a & ind-B to free up disk space just on ind-c
2. Prevent any new data from being indexed to either index on ind-c
1. ssh ind-c
2. stop splunk
3. cd $SPLUNK_DB
4. cd big-index-dir-1 (80GB)
5. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
6. cd big-index-dir-2 (100GB)
7. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
8. du -sh $SPLUNK_DB (Plenty of space now)
9. Do not edit the ~/etc/slave-apps/_cluster/local/indexes.conf (this is controlled by the MN)
10. Instead do this: cd ~/etc/system/local
11. Create a new indexes.conf and insert: