topic Indexer volume unbalanced in Getting Data In

Indexer volume unbalanced

sonicZ — Mon, 29 Oct 2012 18:39:47 GMT

It seems like our indexers do not properly get distributed load in our cluster according to our volume report alerts, it seems rather unbalanced and varies per day.

One example:

> splunk1-d1-inf    18367.6698217417
> splunk2-d2-inf    19339.4671251025
> splunk3-d1-inf    32423.2436867686
> splunk4-d2-inf    19686.7210809250

Another example:

splunk-w1-inf53  13694.9359103119
splunk2-w2-inf54 32902.3739299938

Currently we use a RR record on pairs of intermediate forwarders that then forward to a RR record of our indexers.
At the splunk conf i heard it might be because large data from certain hosts might keep a filehandle open to a specific indexer/intermediate forwarder.

Some people mentioned it be best to use individual IPS in outputs.conf to our intermediate fwrds/indexers instead of our RR DNS records?

Re: Indexer volume unbalanced

sowings — Thu, 01 Nov 2012 02:26:27 GMT

The preferred way is to specify all of the available indexer addresses in the outputs.conf. This allows the forwarder to utilize its load-balancing algorithms to more evenly balance the load. The RR DNS approach tends to develop an affinity between the forwarder and the indexer to which it connects; the forwarder doesn't realize that it needs to be looking for a new host to connect to.

Re: Indexer volume unbalanced

sowings — Thu, 01 Nov 2012 02:28:50 GMT

Re: Indexer volume unbalanced

sonicZ — Fri, 02 Nov 2012 21:59:46 GMT

Sowings thanks for the response and confirm It's what i suspected. we have many agents ill probably have to update them all with puppet since we have no deployment server.