Questions about Splunk Queues.

[Genti]

Why do they become blocked?
How are they related to each other? What is the hierarchy?
What does it mean for a queue to be blocked?
Does increasing queue size help?
How do queues being blocked relate to hardware on the machine?
Is there documentation anywhere on splunk.com where we can find all relevant information in one page?

[bong0004]

Some questions are answered here:

http://wiki.splunk.com/Community:HowIndexingWorks

[Stephen_Sorkin · Accepted solution]

1. Queues become blocked when some (index-time) component in the system can't service data as fast as data is entering the system. For example, if date parsing can only operate at 30K E/s, and we could read data from files at >30K E/s, all upstream queues from date parsing will block.
2. The standard index time queues and processors are: [inputs] -> parsingQueue -> [utf8 processor, line breaker, header parsing] -> aggQueue -> [date parsing and line merging] -> typingQueue -> [regex replacement, punct:: addition] -> indexQueue -> [tcp output, syslog output, http output, block signing, indexing, indexing metrics].
3. If a queue is blocked, that means that data can't be added into the queue and the upstream pipeline will wait until it's empty.
4. No, in steady state changing a queue's size will not materially impact when it gets blocked or system throughput characteristics. Blockage is simply caused by a mismatch of input to output rate.

[hugocvg]

what is the recommendable relation between queues size?

[rroberts]

Is event typing also handled by the typingQueue?

[Genti]

The idea is that i would like an answers page where one can go and get most "general" questions regarding queues in one place, instead of checking multiple answers pages to get the big picture. I know that a doc is being worked on answering these questions and when completed hopefully a link to this page will be given.

[Simeon]

Thats a lot of questions at once... you should break out your questions or have a more direct one.