Where does the persistent queue fit in the data pipeline?

Utkc137 — Thu, 26 Oct 2023 20:02:58 GMT

Was just going through the ‘Masa diagrams’ link: https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774

If you look at the "Detail Diagram - Standalone Splunk" , the queues are laid out like this (one example):

(persistentQueue) + udp_queue --> parsingQueue --> aggQueue --> typingQueue --> indexQueue

So lets say we have UDP input configured and some congestion occurs in typingQueue, the persistentQueue should still be able to hold the data until the congestion is cleared up. This should be able to prevent the data loss.. right?

Sorry for this loaded assumption based question.. I am trying to figure out what we can do to stop UDP input data from getting dropped due to the typingQueue being filled. (P.S. adding an extra HF is not an option right now).

Thanks in advance!

Re: Where does the persistent queue fit in the data pipeline?

gcusello — Fri, 27 Oct 2023 06:59:58 GMT

Hi @Utkc137,

you have three solutions:

use an rsyslog server to receive UDP traffic that writes the logs in a file that's read by a Forwarder, in this case it works also if Splunk is down,
use SC4S app (https://splunkbase.splunk.com/app/4740),
add to your inputs.conf the parameter persistentQueueSize (https://docs.splunk.com/Documentation/Splunk/9.1.1/Admin/Inputsconf) using a large value.

I don't like SC4S, so I hint to use both the the 1st and 3rd solutions.

Ciao.

Giuseppe

topic Where does the persistent queue fit in the data pipeline? in Getting Data In

Where does the persistent queue fit in the data pipeline?

Re: Where does the persistent queue fit in the data pipeline?