How Many Indexes Per Peer (Overall size)

[richardgosnay]

Hey Splunk Friends,

I currently have 32 indexes spread across 2 peers managed by 1 master.  The total space for these indexes has now reached just under 3,000Gb (one of the indexes alone is 1,486Gb).

We don't really have any performance issues at present, but when the Splunk machines get restarted for any reason, it does take some time for the Indexes to catch up (Replication Factor, Search, etc).  On the odd occasion, if there has been an issue which lasted longer, it has caused us to see bucket issues.

My question, is 32 indexes (3000Gb) too much for one cluster (two peers)? If so, should I create another cluster? Or add additional peers?

[s2_splunk · Accepted solution]

It's not so much a question about how many indices you have (within reason). Also, it is the total bucket count in the cluster that contributes the most to any operational processes, like restarts and such. The real benefit of increasing your indexer count would be in being better able to distribute & parallelize the searches your users run against the data, which will likely improve search performance overall. Also, with only two peers in a cluster, the cluster can never return to valid and complete status when you lose a peer (assuming you have RF=2).

You would certainly want to scale your existing cluster vs. creating a second cluster and dealing with the administrative overhead of managing a second cluster manager and ensuring the cluster configurations are identical. Going wider also gives you more replication targets that can receive replicas during/after any peer outages.

For normal/planned restarts, consider putting the cluster into maintenance mode to prevent any fixup attempts (which will fail anyway given you only have two peers).

Finally, there are some significant improvements implemented for the cluster manager in 8.1.x that greatly reduce the time it takes for peers and/or cluster manager to restart, so consider upgrading if you are not already on that version.