Few words about setup that i have: -about 30 indexers in cluster. -following data chain: Syslog Forwarder ( or UF installed on servers) -> Intermediate forwarder(s) at particular sites -> Intermediate Forwarders tier ( 4 IFs machine) - > indexer tier.
When i am checking indexing rate in DMC I can see that no all indexers have similar indexing rate: 4-5 of them have quite indexing rate, rest 15-20 have medium value and rest of them seems to be not used in indexing process.
I want to ask if there is any way how to configure setup in order have balanced value of indexing rate for all indexers, In other words i want to get situation that for most of indexers indexing rate will be at similar level.
Should I use load balancing for achieving this goal ?
There's a few things that could be occurring here, so bare with me.
1) Are all the indexers same spec? RAM, CPU, and most importantly IO.. ( nvme / ssd / sas )
2) Are the search workloads pinning on any of these indexers? Is data balanced on across the indexers?
3) How many intermediates are you funneling the traffic through?
4) What's your autolb setting for the intermediates?
5) How many pipelines do you have on your intermediates?
Going through the above and thinking about Splunk specific configs (assuming equal hardware and sufficient resources..) you may have some bottlenecks with your intermediate tier funneling traffic.
Default autolb frequency is 30 seconds, have you adjusted this? Lowering this to 10s or 5s in large volume environments will help spread data more evenly across the indexing fleet. Additionally, how are the pipelines in your intermediate tier? Rule of thumb is you need 2 X # of indexing pipelines.
So in your case, you should have at least 60 pipelines in your intermediates to get the best event spread across your fleet.
Another point to check is if the internal logs are showing any timeouts, or connection refused, to these indexers in questions..
There are a few starting points.. Let us know how it goes.
I'm totally agreeing with @esix_splunk that you must have enough pipelines on your IHFs (which you probably haven't by default). Without those there haven't been enough events go through those to utilise all indexers at a same time.