It seems like our indexers do not properly get distributed load in our cluster according to our volume report alerts, it seems rather unbalanced and varies per day.
> splunk1-d1-inf 18367.6698217417 > splunk2-d2-inf 19339.4671251025 > splunk3-d1-inf 32423.2436867686 > splunk4-d2-inf 19686.7210809250
splunk-w1-inf53 13694.9359103119 splunk2-w2-inf54 32902.3739299938
Currently we use a RR record on pairs of intermediate forwarders that then forward to a RR record of our indexers.
At the splunk conf i heard it might be because large data from certain hosts might keep a filehandle open to a specific indexer/intermediate forwarder.
Some people mentioned it be best to use individual IPS in outputs.conf to our intermediate fwrds/indexers instead of our RR DNS records?
The preferred way is to specify all of the available indexer addresses in the outputs.conf. This allows the forwarder to utilize its load-balancing algorithms to more evenly balance the load. The RR DNS approach tends to develop an affinity between the forwarder and the indexer to which it connects; the forwarder doesn't realize that it needs to be looking for a new host to connect to.
Sowings thanks for the response and confirm It's what i suspected. we have many agents ill probably have to update them all with puppet since we have no deployment server.