layout:
I have indexer cluster consisting of 3 indexers(ind-a, ind-b, ind-c). Replication and search factor is set to: 3. All works fine, no issues.
Problem:
But, I keep running out of disk space on ind-c only. Following are the steps I was thinking of applying but wanted to make sure this is the right approach.
Concern:
Is that the indexes.conf in ~/etc/slave-apps/_cluster/local will override the indexes.conf in etc/system/local, thus creating a loop and not resolving my issue. Can someone help with my flow
Approach:
1. Select 2 large indexes that I can afford to lose since they are already replicated on ind-a & ind-B to free up disk space just on ind-c
2. Prevent any new data from being indexed to either index on ind-c
Steps:
1. ssh ind-c
2. stop splunk
3. cd $SPLUNK_DB
4. cd big-index-dir-1 (80GB)
5. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
6. cd big-index-dir-2 (100GB)
7. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
8. du -sh $SPLUNK_DB (Plenty of space now)
9. Do not edit the ~/etc/slave-apps/_cluster/local/indexes.conf (this is controlled by the MN)
10. Instead do this: cd ~/etc/system/local
11. Create a new indexes.conf and insert:
[big-index-dir-1]
# turned off
repFactor = 0
homePath = $SPLUNK_DB/big-index-dir-1/db
coldPath = $SPLUNK_DB/big-index-dir-1/colddb
thawedPath = $SPLUNK_DB/big-index-dir-1/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-1/frozen-archivedb
maxTotalDataSizeMB = 80000
frozenTimePeriodInSecs = 259200
[big-index-dir-2]
# turned OFF
repFactor = 0
homePath = $SPLUNK_DB/big-index-dir-2/db
coldPath = $SPLUNK_DB/big-index-dir-2/colddb
thawedPath = $SPLUNK_DB/big-index-dir-2/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-2/frozen-archivedb
maxTotalDataSizeMB = 100000
frozenTimePeriodInSecs = 259200
splunk start
