Getting Data In

Indexer volume unbalanced

sonicZ
Contributor

It seems like our indexers do not properly get distributed load in our cluster according to our volume report alerts, it seems rather unbalanced and varies per day.

One example:

> splunk1-d1-inf    18367.6698217417
> splunk2-d2-inf    19339.4671251025
> splunk3-d1-inf    32423.2436867686
> splunk4-d2-inf    19686.7210809250

Another example:

splunk-w1-inf53  13694.9359103119
splunk2-w2-inf54 32902.3739299938

Currently we use a RR record on pairs of intermediate forwarders that then forward to a RR record of our indexers.
At the splunk conf i heard it might be because large data from certain hosts might keep a filehandle open to a specific indexer/intermediate forwarder.

Some people mentioned it be best to use individual IPS in outputs.conf to our intermediate fwrds/indexers instead of our RR DNS records?

0 Karma
1 Solution

sowings
Splunk Employee
Splunk Employee

The preferred way is to specify all of the available indexer addresses in the outputs.conf. This allows the forwarder to utilize its load-balancing algorithms to more evenly balance the load. The RR DNS approach tends to develop an affinity between the forwarder and the indexer to which it connects; the forwarder doesn't realize that it needs to be looking for a new host to connect to.

View solution in original post

sowings
Splunk Employee
Splunk Employee

The preferred way is to specify all of the available indexer addresses in the outputs.conf. This allows the forwarder to utilize its load-balancing algorithms to more evenly balance the load. The RR DNS approach tends to develop an affinity between the forwarder and the indexer to which it connects; the forwarder doesn't realize that it needs to be looking for a new host to connect to.

sonicZ
Contributor

Sowings thanks for the response and confirm It's what i suspected. we have many agents ill probably have to update them all with puppet since we have no deployment server.

0 Karma

sowings
Splunk Employee
Splunk Employee

See also this answer.

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...