When one indexer in cluster exceeds disk quota, what can be done to keep it from becoming unsearchable?

[mschon314]

I have a 6 indexer cluster (rep factor 3), all 6 indexers are about to fill up on disk space. I added 6 more peers to the cluster, so now there are 12 total indexers.

I disabled forwarding data to the old 6 indexers, only the new 6 indexers will receiving incoming data. But the old 6 will continue to received replicated buckets and are continuing to fill up.

From what I understand, when the disk space on an indexer falls below the quota, default 5GB, the indexers will stop indexing data, and become unsearchable. The unsearchable part is what bothers me. I would much prefer the indexer to remain searchable, but stop indexing. That would make the most sense. But there doesn't seem to be a way to configure that.

So aside, from adding more disk capacity to the old 6 indexers, what can be done to keep indexers from becoming unsearchable?

[mschon314 · Accepted solution]

To answer my own question, the solution is to separate the $SPLUNK_DB directory and the $SPLUNK_HOME/var/run/splunk/dispatch directory onto different disk partitions. Splunk checks that there is enough space for each index directory, and it checks if there is enough space in the dispatch directory. Even symlinking the dispatch directory to another partition works, which is what i've done as a temporary solution. So now on the filled indexer I can run searches, but no new data can be indexed.