Splunk Asynchronous Forwarding Explained

[hrawat]

Splunk asynchronous forwarding is often misunderstood as simply setting autoLBVolume. That is not quite right. The key point is that asynchronous forwarding became possible because the behavior behind autoLBVolume was redesigned starting in Splunk 7.3.6. Before that release, autoLBVolume still existed, but it operated using a synchronous forwarding model.

Important clarification: autoLBVolume is a setting. Asynchronous forwarding is the forwarding behavior enabled by the newer design of that setting in supported versions.

What changed in Splunk 7.3.6

Before Splunk 7.3.6, load-balanced forwarding using autoLBVolume was synchronous. A forwarder would continue sending to its current target in a more serialized way, which meant slower or unhealthy receivers could delay overall forwarding behavior.

Starting with Splunk 7.3.6, autoLBVolume was redesigned to support asynchronous forwarding. This allowed the forwarder to move data more efficiently across multiple receiving indexers or intermediate forwarders, reducing the chance that one slow destination would stall ingestion flow.

Why this matters

• Better data distribution: Data is spread more evenly across receivers.

• Higher ingestion throughput: Forwarders can push data faster in high-volume environments.

• Less impact from slow receivers: A few overloaded indexers are less likely to pause or bottleneck the pipeline.

• Lower intermediate-tier cost: Efficient forwarding can reduce pressure on heavy forwarders used as a routing layer.

Practical outcome: In busy environments, asynchronous forwarding can improve both indexing performance and downstream search performance by keeping bucket and event distribution more balanced.

Common misconception

A common statement is: “Asynchronous forwarding is autoLBVolume.” A more accurate statement is:

autoLBVolume is the mechanism used for load-based forwarding, and from Splunk 7.3.6 onward that mechanism operates asynchronously.

So if someone says “enable async by configuring autoLBVolume,” that is directionally useful, but technically incomplete. The version matters. On versions earlier than 7.3.6, configuring autoLBVolume does not give you the modern asynchronous behavior.

Do not assume feature parity across versions: the same setting name can behave differently depending on the Splunk release.

Version-aware summary

Splunk version	autoLBVolume behavior	Forwarding model
Before 7.3.6	Classic implementation	Synchronous
7.3.6 and later	Redesigned implementation	Asynchronous
Later 9.x releases	Matured support with broader compatibility	Asynchronous

Configuration guidance

When using asynchronous forwarding, autoLBVolume should be sized based on observed throughput, and maxQueueSize should be large enough to support efficient buffering. Connection behavior also matters, especially if acknowledgement is enabled.

• With ACK: use larger queue sizing and longer connection lifetime.

• Without ACK: use a shorter connection lifetime aligned to receiver behavior.

• Avoid mixing with time-based load balancing: forceTimebasedAutoLB=false is important for async behavior.

Hard rule: asynchronous forwarding does not work with forceTimebasedAutoLB. If that setting is enabled, you are not using the intended async forwarding behavior.

How to configure asynchronous forwarding for Splunk 9.x and above

Use the following examples for Splunk Heavy Forwarder 9.x and above. Size autoLBVolume from the forwarder throughput in metrics.log, and size maxQueueSize at least five times larger than autoLBVolume.

Without acknowledgement

[tcpout:XXXXXXXX]
autoLBVolume = <metrics.log average_kbps> * 1000 (minimum 512000)
maxQueueSize = (5 or more) x autoLBVolume (minimum 10MB)
# connectionTTL = Less than receiver shutdown timeout (90 default)
connectionTTL = 75
connectionTimeout = 10
forceTimebasedAutoLB = false

With acknowledgement

[tcpout:XXXXXXXX]
autoLBVolume = <metrics.log average_kbps> * 1000 (minimum 512000)
maxQueueSize = (5 or more) x autoLBVolume (minimum 20MB)
connectionTTL = 300
useACK = true
connectionTimeout = 10
forceTimebasedAutoLB = false

How to configure asynchronous forwarding for Splunk 7.3.6 to 8.x

Use the following examples for Splunk Heavy Forwarder 7.3.6 through 8.x. In these versions, set heartbeatFrequency based on whether acknowledgement is enabled. This setting is not required for Splunk IUF/IHF 9.0 onward.

With acknowledgement

[tcpout:XXXXXXXX]
forceTimebasedAutoLB = false
autoLBVolume = <metrics.log average_kbps> * 1000 (minimum 512000) maxQueueSize = (5 or more) x autoLBVolume (minimum 20MB)
connectionTTL = 300
useACK = true


# heartbeatFrequency Must be less than writeTimeout. Not needed for Splunk IUF/IHF 9.0 onwards

heartbeatFrequency = 15


Without acknowledgement

[tcpout:XXXXXXXX]
forceTimebasedAutoLB = false
autoLBVolume = <metrics.log average_kbps> * 1000 (minimum 512000) maxQueueSize = (5 or more) x autoLBVolume (minimum 10MB)
#connectionTTL = Less than receiver shutdown timeout (90 default)
connectionTTL = 75
# heartbeatFrequency Must be greater than writeTimeout. Not needed for Splunk IUF/IHF 9.0 onward.
heartbeatFrequency = 350


Version note: For Splunk 7.3.6 to 8.x, configure heartbeatFrequency explicitly. For Splunk IUF/IHF 9.0 onward, this setting is no longer required in this async forwarding configuration.

How to calculate autoLBVolume

Calculate autoLBVolume from the average throughput reported in metrics.log. For multiple forwarders, run the SPL across all forwarder hosts, convert average_kbps into bytes, and then average the result.

index=_internal sourcetype=splunkd component=metrics earliest=-24h group=thruput name=thruput host=<all fwd hosts> | eval autoLBVolume = average_kbps * 1024 | stats avg(autoLBVolume)

Rule of thumb: Use the calculated value for autoLBVolume, but do not set it below the minimum recommended value of 512000. Then size maxQueueSize to at least five times autoLBVolume, using the higher minimum for ACK-enabled configurations.

Note: For asynchronous forwarding, keep forceTimebasedAutoLB=false. Do not enable time-based load balancing when using this async forwarding configuration.

Short version

If you want the simplest accurate explanation:

• autoLBVolume existed before 7.3.6.

• Before 7.3.6, it was synchronous.

• From 7.3.6 onward, it was redesigned to be asynchronous.

• So asynchronous forwarding is not just the setting name; it is the post-7.3.6 behavior of that setting.

Reference

• Read the LinkedIn post

[pkiripolsky]

In case anyone is looking for a way to visualize what this looks like, I threw together a query that I use in a dashboard to see how much TCP data each indexer in Splunk Cloud is receiving from a full install of Splunk (i.e Heavy Forwarder).  I haven't tested this in an on-premise environment but this query can likely be adapted.

index=_internal source=*metrics* group=tcpin_connections host="idx-i-*"
[
search index=_internal
sourcetype=splunkd
TERM(group=tcpin_connections)
(TERM("cooked") OR TERM("cookedSSL"))
hostname!=*.splunk*.*
fwdType!="edge"
fwdType!="ueba"
| stats latest(fwdType) AS fwdType BY hostname
| where fwdType="full"
| fields hostname
]
| timechart span=15m sum(kb) limit=0 BY host

If you want to expand it out to Universal Forwarders as well, you can remove:

| where fwdType="full"

Here are the results from the query when Async Forwarding is off:

And here are the results when Async Forwarding is on:

Notice the smoothed out, consistent data flows in the second image.