Deployment Architecture

How do I delete data from just one specific index on one specific indexer in a cluster to free up disk space?

damonmanni
Path Finder

layout:
I have indexer cluster consisting of 3 indexers(ind-a, ind-b, ind-c). Replication and search factor is set to: 3. All works fine, no issues.

Problem:
But, I keep running out of disk space on ind-c only. Following are the steps I was thinking of applying but wanted to make sure this is the right approach.

Concern:
Is that the indexes.conf in ~/etc/slave-apps/_cluster/local will override the indexes.conf in etc/system/local, thus creating a loop and not resolving my issue. Can someone help with my flow

Approach:
1. Select 2 large indexes that I can afford to lose since they are already replicated on ind-a & ind-B to free up disk space just on ind-c
2. Prevent any new data from being indexed to either index on ind-c

Steps:
1. ssh ind-c
2. stop splunk
3. cd $SPLUNK_DB
4. cd big-index-dir-1 (80GB)
5. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
6. cd big-index-dir-2 (100GB)
7. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
8. du -sh $SPLUNK_DB (Plenty of space now)
9. Do not edit the ~/etc/slave-apps/_cluster/local/indexes.conf (this is controlled by the MN)
10. Instead do this: cd ~/etc/system/local
11. Create a new indexes.conf and insert:

[big-index-dir-1]
# turned off 
repFactor  = 0           
homePath   = $SPLUNK_DB/big-index-dir-1/db
coldPath   = $SPLUNK_DB/big-index-dir-1/colddb
thawedPath = $SPLUNK_DB/big-index-dir-1/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-1/frozen-archivedb
maxTotalDataSizeMB = 80000
frozenTimePeriodInSecs = 259200

[big-index-dir-2]
# turned OFF
repFactor  = 0
homePath   = $SPLUNK_DB/big-index-dir-2/db
coldPath   = $SPLUNK_DB/big-index-dir-2/colddb
thawedPath = $SPLUNK_DB/big-index-dir-2/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-2/frozen-archivedb
maxTotalDataSizeMB = 100000
frozenTimePeriodInSecs = 259200

splunk start

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...