https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/Data/Usepersistentqueues
Persistent queuing is available for certain types of inputs, but not all.
One major limitation with persistent queue at inputs enabled on certain UF/HF/IHF/IUF inputs, if downstream parsingqueue/indexqueue/tcpoutqueue are blocked/saturated and a DS bundle push triggers splunk restart, events will be dropped if UF/HF/IHF/IUF failed to drain queues.
On windows DC, persistent queuing is enabled for windows modular inputs, DS bundle push triggers DC restart and events in parsingqueue/tcpoutqueue will be dropped.
On windows DC, some windows event (event occurred while the workstation was being shut down ) logs are always lost.
When Laptops are off the network and restarted/shutdown, in-memory queue events are dropped.
Now new persistent queue is available at tcpoutqueue, it will prevent any kind of loss when splunk restart happens regardless of input type of the event.
Splunk 9.4 ( future release) has added persistent queue at tcpout. Eliminating the need to enable persistent queue on certain supported inputs.
New tcpout persistent queue will solve persistent queue need arising due to unavoidable restarts/ rolling restarts/off network laptops.
Benefits:
It's at no additional cost.
On windows DC, winevent logs are not lost when Laptops are off VPN and are shutdown. Output PQ will write in-memory events on to disk during splunk shutdown.
Forwarders can use splunk useACK( which can be used only between splunk to splunk instances).
No loss of metadata information.
No degraded splunk indexing performance.
Provide persistent queue support for modinputs running on forwarder.
Provide persistent queue support for HEC server running on forwarder.
Writes all in-memory/in-flight events to output persistent queue during splunk restart or a back-pressure.
Instead of paid PQ route
With third party PQ, during splunk restart on forwarding tier, still splunk in-memory queued events might get dropped.
