When a Splunk forwarder loses connectivity to its indexers, it does not always reconnect immediately. In many environments, this looks like a mysterious delay after the network comes back: the path is healthy again, but forwarding resumes only after a waiting period. The reason is backoff.
By default, the maximum backoff can reach 300 seconds because the default autoLBFrequency is 30, and active backoff is calculated as 1-10 * autoLBFrequency.
If your goal is to make forwarders retry almost immediately after a full network outage, the key is not an old backoff toggle. The effective control is autoLBFrequencyIntervalOnGroupFailure.
What causes the delay?
Forwarders apply backoff after repeated write failures to an indexer. The related settings are:
maxFailuresPerInterval = 2
secsInFailureInterval = 1
That means if the forwarder sees more than two failures within one second, it enters backoff. The important part is how long that backoff lasts.
1-10 * autoLBFrequency
This is the active behavior. The backoff starts at 1 * autoLBFrequency, then increases on subsequent failures to 2 * autoLBFrequency, and keeps growing until it is capped at 10 * autoLBFrequency.
With the default value autoLBFrequency = 30, the progression can rise to a maximum of 300 seconds.
If you already use a lower value, the cap is lower too. For example:
Can a forwarder skip backoff entirely?
Do not use backoffOnFailure for this purpose. It is an unused configuration and no longer active.
So if the question is, “Can I disable backoff with backoffOnFailure?” the answer is no. Ignore that setting.
In practice, the forwarder’s retry behavior is driven by the active backoff logic tied to autoLBFrequency, unless group-failure behavior overrides it.
The setting that matters for fast recovery
If the entire target group is unreachable, you can make the forwarder retry much faster by setting:
autoLBFrequencyIntervalOnGroupFailure = <integer>
This setting changes how often the forwarder attempts to reconnect when no target in the tcpout group is reachable.
autoLBFrequencyIntervalOnGroupFailure = <integer>
* When the entire target group is not reachable,
'autoLBFrequencyIntervalOnGroupFailure' is the amount of time, in seconds,
that a forwarder waits before attempting to connect to a target host in the
group.
* While 'autoLBFrequencyIntervalOnGroupFailure' is in effect, 'autoLBFrequency'
is ignored. Once first connection is established to a group, 'autoLBFrequency'
comes into effect again.
* This setting is applied only when
'autoLBFrequencyIntervalOnGroupFailure' is less than 'autoLBFrequency'.
* Every 'autoLBFrequencyIntervalOnGroupFailure' seconds, a new indexer is
selected randomly from the list of indexers provided in the server setting
of the target group stanza.
* -1 means this setting is not active.
* Default: -1
Best practice: If you want near-immediate recovery after a full outage, set autoLBFrequencyIntervalOnGroupFailure to a very low value such as 1.
How it works in real life
autoLBFrequency and autoLBFrequencyIntervalOnGroupFailure are independent, but when the full tcpout group is down, the lower applicable setting governs retry timing.
Here is the practical behavior:
-
If the whole group is unreachable, autoLBFrequencyIntervalOnGroupFailure can take over
-
While it is active, autoLBFrequency is ignored
-
Once one connection in the group succeeds, normal autoLBFrequency behavior resumes
This means you can preserve your normal load-balancing cadence while still making outage recovery aggressive.
Recommended configuration
If you want the forwarder to retry every second after the entire destination group goes down, configure:
[tcpout]
autoLBFrequency = 30
autoLBFrequencyIntervalOnGroupFailure = 1
maxFailuresPerInterval = 2
secsInFailureInterval = 1
With this setup:
-
Normal forwarding behavior still uses autoLBFrequency = 30
-
If the entire tcpout group becomes unreachable, the forwarder retries every 1 second
-
You do not need to reduce autoLBFrequency just to improve outage recovery
Outcome: Setting autoLBFrequencyIntervalOnGroupFailure = 1 effectively reduces maximum wait time after full group failure to about 1 second, without changing your regular autoLBFrequency.
What to remember
-
The default backoff behavior is tied to autoLBFrequency
-
Default maximum backoff is 300 seconds because 10 * 30 = 300
-
backoffOnFailure is unused and should be ignored
-
For fast reconnect after a complete outage, use autoLBFrequencyIntervalOnGroupFailure
-
If you set autoLBFrequencyIntervalOnGroupFailure = 1, the forwarder can retry every second when the whole tcpout group is unavailable
If your goal is “reconnect immediately after the network comes back,” this is the cleanest approach: keep your normal load-balancing settings, and add a low autoLBFrequencyIntervalOnGroupFailure so the forwarder probes the target group aggressively during full outage conditions.
