Getting Data In

When one indexer in cluster exceeds disk quota, what can be done to keep it from becoming unsearchable?

mschon314
Explorer

I have a 6 indexer cluster (rep factor 3), all 6 indexers are about to fill up on disk space. I added 6 more peers to the cluster, so now there are 12 total indexers.

I disabled forwarding data to the old 6 indexers, only the new 6 indexers will receiving incoming data. But the old 6 will continue to received replicated buckets and are continuing to fill up.

From what I understand, when the disk space on an indexer falls below the quota, default 5GB, the indexers will stop indexing data, and become unsearchable. The unsearchable part is what bothers me. I would much prefer the indexer to remain searchable, but stop indexing. That would make the most sense. But there doesn't seem to be a way to configure that.

So aside, from adding more disk capacity to the old 6 indexers, what can be done to keep indexers from becoming unsearchable?

1 Solution

mschon314
Explorer

To answer my own question, the solution is to separate the $SPLUNK_DB directory and the $SPLUNK_HOME/var/run/splunk/dispatch directory onto different disk partitions. Splunk checks that there is enough space for each index directory, and it checks if there is enough space in the dispatch directory. Even symlinking the dispatch directory to another partition works, which is what i've done as a temporary solution. So now on the filled indexer I can run searches, but no new data can be indexed.

View solution in original post

mschon314
Explorer

To answer my own question, the solution is to separate the $SPLUNK_DB directory and the $SPLUNK_HOME/var/run/splunk/dispatch directory onto different disk partitions. Splunk checks that there is enough space for each index directory, and it checks if there is enough space in the dispatch directory. Even symlinking the dispatch directory to another partition works, which is what i've done as a temporary solution. So now on the filled indexer I can run searches, but no new data can be indexed.

Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...