Load Balancing at Splunk

[ibraheem]

With load balancing the Universal Forwarder sends data to all the indexers equally so that no indexer should get all the data and together the indexers holds all the data. It also provide automatic switchover capability incase of an indexer goes down.

Load balancing can be setup at UF in outputs.conf file in two ways:

1. By time
2. By Volume

For time based load balancing we used autoLBFrequency setting and for volume we use autoLBVolume.

Let's say I've three indexers on which I want to send data from UF. My outputs.conf file will look like below:

[tcpout: my_indexers] 
server=10.10.10.1:9997, 10.10.10.2:997, 10.10.10.3:9997

Now, to send data for 3 minutes to an indexer, then switch to another indexer and then to another, set the autoLBFrequency like this:

autoLBFrequency=180

Based on the above settings the UF will send data to indexer 10.10.10.1 for 3 minutes continuously then it will move towards the other indexers, and this loop will continue.

To send data based on the volume. Let's say to configure the UF to send 1MB data to an indexer then switch to another indexer in the list, the setting will look like below

autoLBVolume=1048576

In the cases of a very large file, such as a chatty syslog file, or loading a large amount of historical data, the forwarder may become "stuck" on one indexer, trying to reach EOF before being able to switch to another indexer. To mitigate this, you can use the forceTimebasedAutoLB setting on the forwarder. With this setting, the forwarder does not wait for a safe logical point and instead makes a hard switch to a different indexer every AutoLB cycle.

forceTimebasedAutoLB = true

To guard against loss of data when forwarding to an indexer you can enable indexer acknowledgment capability. With indexer acknowledgment, the forwarder will resend any data that the indexer does not acknowledge as "received". useACK setting is used for this purpose

useACK= true

The final output.conf will look like below

[tcpout]
useACK= true
autoLBFrequency=180
autoLBVolume=1048576

[tcpout: my_indexers] 
server=10.10.10.1:9997, 10.10.10.2:997, 10.10.10.3:9997