Knowledge Management

Splunk TcpOutput Persistent Queue

hrawat_splunk
Splunk Employee
Splunk Employee

https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/Data/Usepersistentqueues

Persistent queuing is available for certain types of inputs, but not all.

One major limitation with persistent queue at inputs  enabled on certain UF/HF/IHF/IUF inputs, if downstream parsingqueue/indexqueue/tcpoutqueue are blocked/saturated and a DS bundle push triggers splunk restart, events will be dropped if UF/HF/IHF/IUF failed to drain queues.

On windows DC, persistent queuing is enabled for windows modular inputs, DS bundle push triggers DC restart and events in parsingqueue/tcpoutqueue will be dropped.

On windows DC, some windows event (event occurred while the workstation was being shut down ) logs are always lost.

When Laptops are off the network and restarted/shutdown, in-memory queue events are dropped. 

Now new persistent queue is available at tcpoutqueue, it will prevent any kind of loss when splunk restart happens regardless of input type of the event. 

Splunk 9.4 ( future release) has added persistent queue at tcpout. Eliminating the need to enable persistent queue on certain supported inputs.

New tcpout persistent queue will solve persistent queue need arising due to unavoidable  restarts/ rolling restarts/off network laptops.

Benefits:
It's at no additional cost.
On windows DC, winevent logs are not lost when Laptops are off VPN and are shutdown. Output PQ will write in-memory events on to disk during splunk shutdown.
Forwarders can use splunk useACK( which can be used only between splunk to splunk instances). 
No loss of metadata information.
No degraded splunk indexing performance.
Provide persistent queue support for modinputs running on forwarder.
Provide persistent queue support for HEC server running on forwarder.
Writes all in-memory/in-flight events to output persistent queue during splunk restart or a back-pressure.


New Splunk output persistent queue.png
Instead of paid PQ route 

hrawat_splunk_0-1716497147444.png

With third party PQ, during splunk restart on forwarding tier, still splunk in-memory queued events might get dropped. 

Labels (1)
Tags (1)
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...