Getting Data In

Indexer volume unbalanced

sonicZ
Contributor

It seems like our indexers do not properly get distributed load in our cluster according to our volume report alerts, it seems rather unbalanced and varies per day.

One example:

> splunk1-d1-inf    18367.6698217417
> splunk2-d2-inf    19339.4671251025
> splunk3-d1-inf    32423.2436867686
> splunk4-d2-inf    19686.7210809250

Another example:

splunk-w1-inf53  13694.9359103119
splunk2-w2-inf54 32902.3739299938

Currently we use a RR record on pairs of intermediate forwarders that then forward to a RR record of our indexers.
At the splunk conf i heard it might be because large data from certain hosts might keep a filehandle open to a specific indexer/intermediate forwarder.

Some people mentioned it be best to use individual IPS in outputs.conf to our intermediate fwrds/indexers instead of our RR DNS records?

0 Karma
1 Solution

sowings
Splunk Employee
Splunk Employee

The preferred way is to specify all of the available indexer addresses in the outputs.conf. This allows the forwarder to utilize its load-balancing algorithms to more evenly balance the load. The RR DNS approach tends to develop an affinity between the forwarder and the indexer to which it connects; the forwarder doesn't realize that it needs to be looking for a new host to connect to.

View solution in original post

sowings
Splunk Employee
Splunk Employee

The preferred way is to specify all of the available indexer addresses in the outputs.conf. This allows the forwarder to utilize its load-balancing algorithms to more evenly balance the load. The RR DNS approach tends to develop an affinity between the forwarder and the indexer to which it connects; the forwarder doesn't realize that it needs to be looking for a new host to connect to.

sonicZ
Contributor

Sowings thanks for the response and confirm It's what i suspected. we have many agents ill probably have to update them all with puppet since we have no deployment server.

0 Karma

sowings
Splunk Employee
Splunk Employee

See also this answer.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...