topic Re: Questions about Splunk Queues. in Getting Data In

Questions about Splunk Queues.

Genti — Wed, 22 Sep 2010 02:29:13 GMT

Why do they become blocked?
How are they related to each other? What is the hierarchy?
What does it mean for a queue to be blocked?
Does increasing queue size help?
How do queues being blocked relate to hardware on the machine?
Is there documentation anywhere on splunk.com where we can find all relevant information in one page?

Re: Questions about Splunk Queues.

Simeon — Wed, 22 Sep 2010 02:36:25 GMT

Thats a lot of questions at once... you should break out your questions or have a more direct one.

Re: Questions about Splunk Queues.

Genti — Wed, 22 Sep 2010 02:41:46 GMT

The idea is that i would like an answers page where one can go and get most "general" questions regarding queues in one place, instead of checking multiple answers pages to get the big picture. I know that a doc is being worked on answering these questions and when completed hopefully a link to this page will be given.

Re: Questions about Splunk Queues.

Stephen_Sorkin — Wed, 22 Sep 2010 10:15:54 GMT

Queues become blocked when some (index-time) component in the system can't service data as fast as data is entering the system. For example, if date parsing can only operate at 30K E/s, and we could read data from files at >30K E/s, all upstream queues from date parsing will block.
The standard index time queues and processors are: [inputs] -> parsingQueue -> [utf8 processor, line breaker, header parsing] -> aggQueue -> [date parsing and line merging] -> typingQueue -> [regex replacement, punct:: addition] -> indexQueue -> [tcp output, syslog output, http output, block signing, indexing, indexing metrics].
If a queue is blocked, that means that data can't be added into the queue and the upstream pipeline will wait until it's empty.
No, in steady state changing a queue's size will not materially impact when it gets blocked or system throughput characteristics. Blockage is simply caused by a mismatch of input to output rate.

Re: Questions about Splunk Queues.

hugocvg — Tue, 19 Feb 2013 02:01:34 GMT

what is the recommendable relation between queues size?

Re: Questions about Splunk Queues.

rroberts — Thu, 22 Jan 2015 21:00:17 GMT

Is event typing also handled by the typingQueue?

Re: Questions about Splunk Queues.

bong0004 — Thu, 30 Apr 2015 20:02:23 GMT

Some questions are answered here:

http://wiki.splunk.com/Community:HowIndexingWorks