Deployment Architecture

Splunk Forwarder AutoLB

gdavid
Path Finder

How does autolb work on a splunk forwarder.
I have a forwarder with outputs.conf pointing at 2 indexers.
When i look at most recent events on each indexer i see they are mostly sitting on the first indexer in my group. This sucks because for most of my queries its coming all off 1 indexer

[tcpout]
maxQueueSize = 500KB
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = _audit
forwardedindex.filter.disable = false
defaultGroup = my-indexers_9997
disabled = false
indexAndForward = 0

[tcpout:my-indexers_9997]
autoLB = true
server = indexer1:9997,indexer2:9997

Do i need to configure:
forceTimebasedAutoLB = [true|false]
* Will force existing streams to switch to newly elected indexer every AutoLB cycle.
* Defaults to false
?

thanks

Gd

khourihan_splun
Splunk Employee
Splunk Employee

See this blog entry to see how forceTimebasedAutoLB=true works.

Essentially if a UF has six events: 1,2,3,4,5,6 to send to two indexers I1 I2

and UF is sending events 1,2,3 to I1, then the LB timer expires halfway through event 3 the UF will send the event halfway, then tag it as incomplete. Then resend another tag, then events 3,4,5,6.

 1,2,3,4,5,6  UF _______/>I1 (1.2.3.5<tag>)
                        \>I2 (<tag>3.5,3,4,5,6)

The tags identify to I1 that the last event is incomplete and to disregard, and to I2, that its getting a partially event (half of 3), then the full event 3, followed by 3,4,5,6.

Probably a little too much info, but the tags help Splunk reassemble the data, and avoid duplicates.

Thanks to Deep for whiteboarding this for me.

rgaleone1
Path Finder

gdavid - Are you asking if changing the default of 30 second down to say 15 seconds before switching indexes is good practice? I am confused what you mean about fail-over time, could you please clarify?

0 Karma

gdavid
Path Finder

standard auto LB seems to work when the stream of data isn't that heavy. i'm assuming that's why it doesn't work so well with syslog from 50+ devices.

i set the forceTimebasedAutoLB = true

now it changes indexers every 30 seconds which i think it a little overkill. my searches are usually over 15 mins spans so i could extend the interval to maybe 3 or 5 mins.

basically my new question is what affect does changing the 30 second interval up to 3+ mins?

if my indexer1 goes down, will it take 3+ mins for it to switch to indexer2?

0 Karma

khourihan_splun
Splunk Employee
Splunk Employee

Gosh, sorry its almost two years later.

If the indexer goes down, the forwarder would try the next one. SplunkTCP is reliable and uses TCP acks so if the connection goes down, Splunk UF's would just try the next one.

Also, we've got customers that set the interval to 10 and even 5 seconds. The advantage there is that if you search across last 1 minute, the data resides on more than one indexer, and the map reduce works better/faster.

Best,
Kyle

0 Karma

gdavid
Path Finder

so after setting forceTimebasedAutoLB=true, i see the syslog data load balanced between the indexers now. its on the default 30 second rotation. i'm worried about increasing that frequency because i believe that will also be the fail-over time in the event of a indexer outage?

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Take Action Automatically on Splunk Alerts with Red Hat Ansible Automation Platform

 Are you ready to revolutionize your IT operations? As digital transformation accelerates, the demand for ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...