Deployment Architecture

Not all indexers are indexing with the similar indexing rate

Path Finder

Hello everyone

Few words about setup that i have:
-about 30 indexers in cluster.
-following data chain: Syslog Forwarder ( or UF installed on servers) -> Intermediate forwarder(s) at particular sites -> Intermediate Forwarders tier ( 4 IFs machine) - > indexer tier.

When i am checking indexing rate in DMC I can see that no all indexers have similar indexing rate:
4-5 of them have quite indexing rate, rest 15-20 have medium value and rest of them seems to be not used in indexing process.

I want to ask if there is any way how to configure setup in order have balanced value of indexing rate  for all indexers, In other words i want to get situation that for most of indexers indexing rate will be at similar level.

Should I use load balancing for achieving  this goal ?


Labels (1)
0 Karma

Splunk Employee
Splunk Employee

There's a few things that could be occurring here, so bare with me.


1) Are all the indexers same spec? RAM, CPU, and most importantly IO.. ( nvme / ssd / sas )

2) Are the search workloads pinning on any of these indexers? Is data balanced on across the indexers?

3) How many intermediates are you funneling the traffic through?

4) What's your autolb setting for the intermediates?

5) How many pipelines do you have on your intermediates?


Going through the above and thinking about Splunk specific configs (assuming equal hardware and sufficient resources..) you may have some bottlenecks with your intermediate tier funneling traffic.

Default autolb frequency is 30 seconds, have you adjusted this? Lowering this to 10s or 5s in large volume environments will help spread data more evenly across the indexing fleet.  Additionally, how are the pipelines in your intermediate tier? Rule of thumb is you need 2 X # of indexing pipelines.

So in your case, you should have at least 60 pipelines in your intermediates to get the best event spread across your fleet. 

Another point to check is if the internal logs are showing any timeouts, or connection refused, to these indexers in questions.. 


There are a few starting points.. Let us know how it goes.

0 Karma



Here is a excellent presentation about data distribution over indexers.

I'm totally agreeing with @esix_splunk that you must have enough pipelines on your IHFs (which you probably haven't by default). Without those there haven't been enough events go through those to utilise all indexers at a same time.

r. Ismo

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...