Getting Data In

Where does the persistent queue fit in the data pipeline?

Utkc137
Explorer

Was just going through the ‘Masa diagrams’ link: https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platfor...

If you look at the "Detail Diagram - Standalone Splunk" , the queues are laid out like this (one example):

(persistentQueue) + udp_queue --> parsingQueue --> aggQueue --> typingQueue --> indexQueue 

So lets say we have UDP input configured and some congestion occurs in typingQueue, the persistentQueue should still be able to hold the data until the congestion is cleared up. This should be able to prevent the data loss.. right?

 Sorry for this loaded assumption based question.. I am trying to figure out what we can do to stop UDP input data from getting dropped due to the typingQueue being filled. (P.S. adding an extra HF is not an option right now).

 

Thanks in advance!

Labels (2)
Tags (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Utkc137,

you have three solutions:

I don't like SC4S, so I hint to use both the the 1st and 3rd solutions.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...