Splunk Enterprise

How Many Indexes Per Peer (Overall size)

richardgosnay
Explorer

Hey Splunk Friends,

 

I currently have 32 indexes spread across 2 peers managed by 1 master.  The total space for these indexes has now reached just under 3,000Gb (one of the indexes alone is 1,486Gb).

We don't really have any performance issues at present, but when the Splunk machines get restarted for any reason, it does take some time for the Indexes to catch up (Replication Factor, Search, etc).  On the odd occasion, if there has been an issue which lasted longer, it has caused us to see bucket issues.

My question, is 32 indexes (3000Gb) too much for one cluster (two peers)? If so, should I create another cluster? Or add additional peers?

Labels (3)
0 Karma
1 Solution

s2_splunk
Splunk Employee
Splunk Employee

It's not so much a question about how many indices you have (within reason). Also, it is the total bucket count in the cluster that contributes the most to any operational processes, like restarts and such. The real benefit of increasing your indexer count would be in being better able to distribute & parallelize the searches your users run against the data, which will likely improve search performance overall. Also, with only two peers in a cluster, the cluster can never return to valid and complete status when you lose a peer (assuming you have RF=2).

You would certainly want to scale your existing cluster vs. creating a second cluster and dealing with the administrative overhead of managing a second cluster manager and ensuring the cluster configurations are identical. Going wider also gives you more replication targets that can receive replicas during/after any peer outages.

For normal/planned restarts, consider putting the cluster into maintenance mode to prevent any fixup attempts (which will fail anyway given you only have two peers).

Finally, there are some significant improvements implemented for the cluster manager in 8.1.x that greatly reduce the time it takes for peers and/or cluster manager to restart, so consider upgrading if you are not already on that version.

View solution in original post

s2_splunk
Splunk Employee
Splunk Employee

It's not so much a question about how many indices you have (within reason). Also, it is the total bucket count in the cluster that contributes the most to any operational processes, like restarts and such. The real benefit of increasing your indexer count would be in being better able to distribute & parallelize the searches your users run against the data, which will likely improve search performance overall. Also, with only two peers in a cluster, the cluster can never return to valid and complete status when you lose a peer (assuming you have RF=2).

You would certainly want to scale your existing cluster vs. creating a second cluster and dealing with the administrative overhead of managing a second cluster manager and ensuring the cluster configurations are identical. Going wider also gives you more replication targets that can receive replicas during/after any peer outages.

For normal/planned restarts, consider putting the cluster into maintenance mode to prevent any fixup attempts (which will fail anyway given you only have two peers).

Finally, there are some significant improvements implemented for the cluster manager in 8.1.x that greatly reduce the time it takes for peers and/or cluster manager to restart, so consider upgrading if you are not already on that version.

Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...