precedence in outputs.conf on heavy forwarders

[Esky73]

I have 2 heavy forwarders that forward to 2 peer indexers their config is identical like so :

[tcpout]
defaultGroup=splunk_cluster_1
maxQueueSize=7MB

[tcpout:splunk_cluster_1]
autoLBFrequency=40
server=x.x.x.98:9997,x.x.x.99:9997
useACK=true

When i made a change on the master node and pushed it out to bundle some alerts were triggered as there was no data for over 2 minutes .. i've been looking into possible reasons. looking at the splunkd logs when one of the indexers restarts then i would expect the heavy forwarder to connect to the indexer thats not restarting. so if both indexers have the same servers listed in the same order - will they try and connect to the first one listed (wait for the frequency = 40s) and try the next one ? so wouldnt it be better to change the servers round on the second instance if that is the case ?

[esix_splunk · Accepted solution]

If your outputs are identical, the HF should autobalance over to the available indexers. When Splunk starts, it will reach out to the server is chooses from outputs. If it doesnt make contact (no ack, no reply), it will go to the next in the list. It wont wait 40seconds before switching.

From my understanding when Splunk starts, it will take a random host from the outputs and start with it, not the first in the list. Or if you are using indexer discovery, it will work the same way with the list returned from the Cluster Master...

Bit more reading : http://docs.splunk.com/Documentation/Splunk/6.4.3/Forwarding/Setuploadbalancingd