Deployment Architecture

How do I delete data from just one specific index on one specific indexer in a cluster to free up disk space?

damonmanni
Path Finder

layout:
I have indexer cluster consisting of 3 indexers(ind-a, ind-b, ind-c). Replication and search factor is set to: 3. All works fine, no issues.

Problem:
But, I keep running out of disk space on ind-c only. Following are the steps I was thinking of applying but wanted to make sure this is the right approach.

Concern:
Is that the indexes.conf in ~/etc/slave-apps/_cluster/local will override the indexes.conf in etc/system/local, thus creating a loop and not resolving my issue. Can someone help with my flow

Approach:
1. Select 2 large indexes that I can afford to lose since they are already replicated on ind-a & ind-B to free up disk space just on ind-c
2. Prevent any new data from being indexed to either index on ind-c

Steps:
1. ssh ind-c
2. stop splunk
3. cd $SPLUNK_DB
4. cd big-index-dir-1 (80GB)
5. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
6. cd big-index-dir-2 (100GB)
7. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
8. du -sh $SPLUNK_DB (Plenty of space now)
9. Do not edit the ~/etc/slave-apps/_cluster/local/indexes.conf (this is controlled by the MN)
10. Instead do this: cd ~/etc/system/local
11. Create a new indexes.conf and insert:

[big-index-dir-1]
# turned off 
repFactor  = 0           
homePath   = $SPLUNK_DB/big-index-dir-1/db
coldPath   = $SPLUNK_DB/big-index-dir-1/colddb
thawedPath = $SPLUNK_DB/big-index-dir-1/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-1/frozen-archivedb
maxTotalDataSizeMB = 80000
frozenTimePeriodInSecs = 259200

[big-index-dir-2]
# turned OFF
repFactor  = 0
homePath   = $SPLUNK_DB/big-index-dir-2/db
coldPath   = $SPLUNK_DB/big-index-dir-2/colddb
thawedPath = $SPLUNK_DB/big-index-dir-2/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-2/frozen-archivedb
maxTotalDataSizeMB = 100000
frozenTimePeriodInSecs = 259200

splunk start

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...