Queues become blocked when some (index-time) component in the system can't service data as fast as data is entering the system. For example, if date parsing can only operate at 30K E/s, and we could read data from files at >30K E/s, all upstream queues from date parsing will block.
The standard index time queues and processors are: [inputs] -> parsingQueue -> [utf8 processor, line breaker, header parsing] -> aggQueue -> [date parsing and line merging] -> typingQueue -> [regex replacement, punct:: addition] -> indexQueue -> [tcp output, syslog output, http output, block signing, indexing, indexing metrics].
If a queue is blocked, that means that data can't be added into the queue and the upstream pipeline will wait until it's empty.
No, in steady state changing a queue's size will not materially impact when it gets blocked or system throughput characteristics. Blockage is simply caused by a mismatch of input to output rate.
... View more