Deployment Architecture

How do I delete data from just one specific index on one specific indexer in a cluster to free up disk space?

damonmanni
Path Finder

layout:
I have indexer cluster consisting of 3 indexers(ind-a, ind-b, ind-c). Replication and search factor is set to: 3. All works fine, no issues.

Problem:
But, I keep running out of disk space on ind-c only. Following are the steps I was thinking of applying but wanted to make sure this is the right approach.

Concern:
Is that the indexes.conf in ~/etc/slave-apps/_cluster/local will override the indexes.conf in etc/system/local, thus creating a loop and not resolving my issue. Can someone help with my flow

Approach:
1. Select 2 large indexes that I can afford to lose since they are already replicated on ind-a & ind-B to free up disk space just on ind-c
2. Prevent any new data from being indexed to either index on ind-c

Steps:
1. ssh ind-c
2. stop splunk
3. cd $SPLUNK_DB
4. cd big-index-dir-1 (80GB)
5. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
6. cd big-index-dir-2 (100GB)
7. rm -rf ./frozen/* ./db/* ./colddb/* ./thawed/*
8. du -sh $SPLUNK_DB (Plenty of space now)
9. Do not edit the ~/etc/slave-apps/_cluster/local/indexes.conf (this is controlled by the MN)
10. Instead do this: cd ~/etc/system/local
11. Create a new indexes.conf and insert:

[big-index-dir-1]
# turned off 
repFactor  = 0           
homePath   = $SPLUNK_DB/big-index-dir-1/db
coldPath   = $SPLUNK_DB/big-index-dir-1/colddb
thawedPath = $SPLUNK_DB/big-index-dir-1/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-1/frozen-archivedb
maxTotalDataSizeMB = 80000
frozenTimePeriodInSecs = 259200

[big-index-dir-2]
# turned OFF
repFactor  = 0
homePath   = $SPLUNK_DB/big-index-dir-2/db
coldPath   = $SPLUNK_DB/big-index-dir-2/colddb
thawedPath = $SPLUNK_DB/big-index-dir-2/thaweddb
coldToFrozenDir = $SPLUNK_DB/big-index-dir-2/frozen-archivedb
maxTotalDataSizeMB = 100000
frozenTimePeriodInSecs = 259200

splunk start

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...