Getting Data In
Highlighted

Indexer volume unbalanced

Contributor

It seems like our indexers do not properly get distributed load in our cluster according to our volume report alerts, it seems rather unbalanced and varies per day.

One example:

> splunk1-d1-inf    18367.6698217417
> splunk2-d2-inf    19339.4671251025
> splunk3-d1-inf    32423.2436867686
> splunk4-d2-inf    19686.7210809250

Another example:

splunk-w1-inf53  13694.9359103119
splunk2-w2-inf54 32902.3739299938

Currently we use a RR record on pairs of intermediate forwarders that then forward to a RR record of our indexers.
At the splunk conf i heard it might be because large data from certain hosts might keep a filehandle open to a specific indexer/intermediate forwarder.

Some people mentioned it be best to use individual IPS in outputs.conf to our intermediate fwrds/indexers instead of our RR DNS records?

0 Karma
Highlighted

Re: Indexer volume unbalanced

Splunk Employee
Splunk Employee

The preferred way is to specify all of the available indexer addresses in the outputs.conf. This allows the forwarder to utilize its load-balancing algorithms to more evenly balance the load. The RR DNS approach tends to develop an affinity between the forwarder and the indexer to which it connects; the forwarder doesn't realize that it needs to be looking for a new host to connect to.

View solution in original post

Highlighted

Re: Indexer volume unbalanced

Splunk Employee
Splunk Employee

See also this answer.

0 Karma
Highlighted

Re: Indexer volume unbalanced

Contributor

Sowings thanks for the response and confirm It's what i suspected. we have many agents ill probably have to update them all with puppet since we have no deployment server.

0 Karma