topic Splunk Forwarder AutoLB in Deployment Architecture

Splunk Forwarder AutoLB

gdavid — Mon, 28 Sep 2020 15:00:14 GMT

How does autolb work on a splunk forwarder.
I have a forwarder with outputs.conf pointing at 2 indexers.
When i look at most recent events on each indexer i see they are mostly sitting on the first indexer in my group. This sucks because for most of my queries its coming all off 1 indexer

[tcpout]
maxQueueSize = 500KB
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = _audit
forwardedindex.filter.disable = false
defaultGroup = my-indexers_9997
disabled = false
indexAndForward = 0

[tcpout:my-indexers_9997]
autoLB = true
server = indexer1:9997,indexer2:9997

Do i need to configure:
forceTimebasedAutoLB = [true|false]
* Will force existing streams to switch to newly elected indexer every AutoLB cycle.
* Defaults to false
?

thanks

Re: Splunk Forwarder AutoLB

gdavid — Fri, 18 Oct 2013 17:42:49 GMT

so after setting forceTimebasedAutoLB=true, i see the syslog data load balanced between the indexers now. its on the default 30 second rotation. i'm worried about increasing that frequency because i believe that will also be the fail-over time in the event of a indexer outage?

Re: Splunk Forwarder AutoLB

rgaleone1 — Fri, 18 Oct 2013 17:56:42 GMT

gdavid - Are you asking if changing the default of 30 second down to say 15 seconds before switching indexes is good practice? I am confused what you mean about fail-over time, could you please clarify?

Re: Splunk Forwarder AutoLB

gdavid — Fri, 18 Oct 2013 18:01:22 GMT

standard auto LB seems to work when the stream of data isn't that heavy. i'm assuming that's why it doesn't work so well with syslog from 50+ devices.

i set the forceTimebasedAutoLB = true

now it changes indexers every 30 seconds which i think it a little overkill. my searches are usually over 15 mins spans so i could extend the interval to maybe 3 or 5 mins.

basically my new question is what affect does changing the 30 second interval up to 3+ mins?

if my indexer1 goes down, will it take 3+ mins for it to switch to indexer2?

Re: Splunk Forwarder AutoLB

khourihan_splun — Tue, 05 Aug 2014 00:34:35 GMT

See this blog entry to see how forceTimebasedAutoLB=true works.

Essentially if a UF has six events: 1,2,3,4,5,6 to send to two indexers I1 I2

and UF is sending events 1,2,3 to I1, then the LB timer expires halfway through event 3 the UF will send the event halfway, then tag it as incomplete. Then resend another tag, then events 3,4,5,6.

 1,2,3,4,5,6  UF _______/>I1 (1.2.3.5<tag>)
                        \>I2 (<tag>3.5,3,4,5,6)

The tags identify to I1 that the last event is incomplete and to disregard, and to I2, that its getting a partially event (half of 3), then the full event 3, followed by 3,4,5,6.

Probably a little too much info, but the tags help Splunk reassemble the data, and avoid duplicates.

Thanks to Deep for whiteboarding this for me.

Re: Splunk Forwarder AutoLB

khourihan_splun — Fri, 07 Aug 2015 06:33:20 GMT

Gosh, sorry its almost two years later.

If the indexer goes down, the forwarder would try the next one. SplunkTCP is reliable and uses TCP acks so if the connection goes down, Splunk UF's would just try the next one.

Also, we've got customers that set the interval to 10 and even 5 seconds. The advantage there is that if you search across last 1 minute, the data resides on more than one indexer, and the map reduce works better/faster.

Best,
Kyle