Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in recent releases. It solves a common operational problem: what happens when forwarding must continue, but the receiving tier is unavailable or unreachable for an extended period? Instead of depending on external queueing systems, the forwarder can now persist outbound events locally on disk and resume transmission when connectivity returns.
Why this feature stands out: it addresses real outages, real maintenance windows, and real disconnected environments with a simple native control in outputs.conf.
- What it is
- Why it matters in the real world
- Data flow during outage and recovery
- Why native PQ is often better than external buffering
- How to enable it
- Sizing guidance
- Critical operational update: the PQ drain memory issue
- Practical design guidance
- Related tuning context
- When this feature is the right fit
- Bottom line
- References
What it is
TcpOut Persistent Queue allows a forwarder or intermediate heavy forwarder to write outbound data to disk when the destination cannot accept data immediately. When the receiving side becomes available again, the queued data is sent onward without requiring a third-party buffer such as SQS or S3.
In practical terms, this gives Splunk administrators a native, local, and operationally simpler buffering option for disruptions that are temporary but too long for in-memory queues alone.
Why it matters in the real world
The strongest value of this feature is not theoretical architecture elegance. It is operational survival.
-
Ships and cruises at sea: a vessel can lose internet or satellite connectivity for days. Logs and telemetry still continue to generate onboard. With tcpout persistent queue, that data can be retained locally and forwarded once connectivity returns.
-
Corporate laptops: systems may stay off VPN or off network for extended periods. Events can be preserved and sent later when the endpoint reconnects.
-
Indexer clustering maintenance: rolling restarts, planned upgrades, and temporary cluster-side slowness no longer have to immediately translate into data loss or emergency queue workarounds.
-
Sensitive environments: data can remain within the managed network boundary instead of being detoured through external services.
-
Multiple tcpout destinations: if one destination is down but has a sufficiently large persistent queue, another active destination can continue processing until the failed path's queue fills.
Best example: disconnected maritime environments are an ideal use case. A ship continues to produce operational, security, and observability data while offline, then automatically drains the backlog when the link is restored.
Data flow during outage and recovery
The following diagram shows the most practical flow using the ships or cruises scenario.
Important caution:
If the backlog drains from disk faster than tcpout can send over the network, memory can spike.
Until fixed versions are in place, size PQ conservatively relative to available RAM.Why native PQ is often better than external buffering
Before this capability, some teams considered third-party buffering patterns for long disconnections. Native tcpout PQ changes that calculation significantly.
-
No additional subscription cost for external queueing services.
-
No permanent detour latency introduced by writing every event out to an external queue first.
-
No dependency on payload size constraints imposed by third-party services.
-
No extra upload and download bandwidth for every event.
-
No dependency on internet connectivity for buffering, because third-party options such as SQS or S3 still require an available network path to the cloud before they can store data.
-
Reduced operational complexity compared with maintaining external queueing, batching, retry logic, and dead-letter handling.
-
Better data locality for environments where sensitive data should remain internal.
This does not mean external architectures never have a place. It means many common Splunk forwarding scenarios can now be solved more cleanly inside the platform.
How to enable it
Enable the feature by setting persistentQueueSize in outputs.conf for the tcpout target group.
[tcpout:splunk-group1]
persistentQueueSize=1TB
[tcpout:splunk-group2]
persistentQueueSize=2TBThis simplicity is part of the appeal. The feature is easy to understand and easy to turn on, provided sizing is thought through carefully.
Sizing guidance
A practical sizing approach is to estimate how much data a forwarding tier sends during the outage window you need to survive. One recommended method is to calculate this from internal metrics.
index=_internal source=*metrics.log* group=tcpin_connections hostname=<all IHF> host=<all idx>
| stats sum(kb) as required_pq_in_kb by hostnameRun the search across the number of hours or days you want to protect. The resulting required_pq_in_kb is the approximate overall PQ needed for a given intermediate heavy forwarder.
A practical approximation from the source material is:
persistentQueueSize = required_pq_in_kb / parallelIngestionPipelinesThis provides a starting point, not a final truth. Real sizing should also consider growth bursts, recovery behavior, disk availability, and memory safety.
Critical operational update: the PQ drain memory issue
Important: an issue was identified where tcpout can read from the persistent queue at unbounded speed after the indexing tier becomes available, even though it cannot push events over TCP at the same rate. This can cause memory spikes and potentially an out-of-memory condition.
The practical implication is straightforward: a very large persistent queue is not automatically a safe persistent queue.
Example from the source material:
-
If a system has 12 GB RAM and
persistentQueueSize=18GB, it is likely to hit OOM if the queue fills or holds more than 12 GB of data. -
If the same system has 12 GB RAM and
persistentQueueSize=8GBor less, it is much less likely to hit the issue.
Interim guidance: until fixed versions are installed, keep persistentQueueSize below the average available memory of the system.
Fix is available in versions 9.4.13, 10.0.7, 10.2.4, and 10.4.1.
Practical design guidance
-
Use tcpout PQ for real disconnect scenarios, not as a substitute for weak capacity planning.
-
Size for the outage you truly expect, especially in ships, offshore systems, mobile fleets, and remote sites.
-
Validate available disk and memory together; disk capacity alone is not enough.
-
Test recovery behavior by simulating outage and reconnection in a lower environment.
-
Review parallel ingestion and queue automation settings on newer Splunk versions where queue behavior is increasingly auto-adjusted.
-
Be conservative on heavy forwarders with bursty HEC input, because sudden backlog release can amplify memory pressure.
Related tuning context
Persistent queueing at tcpout is most useful when viewed as part of a broader ingestion resilience strategy. Related operational guidance from internal documentation emphasizes that newer Splunk releases increasingly automate indexing and queue behavior, including settings such as autoAdjustQueue=true and queue sizing improvements on modern versions.
For on-prem environments experiencing full replication or ingestion pressure, queue and pipeline tuning may still matter. However, tcpout PQ is especially valuable on the forwarding side because it provides a native answer to sudden spikes, disconnected operation, and upstream outage windows.
When this feature is the right fit
|
Scenario |
Why tcpout PQ helps |
Main design concern |
|---|---|---|
|
Ships, cruises, offshore systems |
Stores data locally during multi-day connectivity loss and forwards later |
Disk sizing for outage duration and safe drain after reconnection |
|
Corporate laptops off VPN |
Retains events while endpoints are disconnected |
Endpoint storage limits and intermittent shutdown behavior |
|
Indexer rolling restart or upgrade |
Buffers forwarding traffic during planned receiver unavailability |
Recovery surge after the receiving tier returns |
|
Temporary indexing slowness |
Prevents immediate forwarding blockage during short-term backpressure |
PQ should not hide chronic capacity shortfalls |
Bottom line
TcpOut Persistent Queue is one of Splunk's most practical features because it solves a problem administrators face constantly: data does not stop just because connectivity does. For maritime systems, remote environments, laptops, and planned indexer-side outages, native outbound persistence can dramatically simplify architecture and improve resilience.
The feature is powerful, easy to enable, and operationally meaningful. The one caution is equally important: size it responsibly, especially on versions affected by the drain-related memory spike issue.
Recommended approach: enable the feature for clearly defined outage scenarios, size it from observed throughput, test drain behavior, and patch to versions that address the memory risk.
persistentQueueSize from observed data volumeReferences
