Using LWF as intermediate forwarder and using autoLB

[gkanapathy]

I understand that auto load-balancing (autoLB) on a Splunk Light Forwarder works by switching indexers for a source only when it reads the end of a monitored source file, to ensure that it only switches between events.

If I use a Light Forwarder as an relay or intermediate forwarder between other light forwarders and a cluster of indexers (because of network restrictions), will autoLB still work? That is, is autoLB dependent on the LWF having the file locally monitored? Or will autoLB still work if the LWF receives the input stream from another set of LWFs? Will it be able to look at the incoming source keys and switch only on the "done" key in the stream, or does autoLB not work that way?

[Stephen_Sorkin · Accepted solution]

AutoLB is keyed just on the done key, which is propagated from the initial to intermediate forwarders. However, in investigating this, we've observed a bug that will keep this from working as expected. Specifically we're keying the stream on just the value of the "source" key rather than an unambiguous representation of the stream. This doesn't affect a single tier of autoLB LWFs, but will cause a second tier of these to inappropriately terminate a connection when a done key is seen for a similarly named source from a different stream. This will be fixed in 4.2.

[gkanapathy]

yes, it would work, but i am concerned about the performance/throughput of using a heavy forwarder vs light, and the number of forwarders i would need to handle peak traffic loads.

[hacktastic]

Is the workaround to use a HWF as an intermediate forwarder?