Getting Data In

Questions about Splunk Queues.

Genti
Splunk Employee
Splunk Employee

Why do they become blocked?
How are they related to each other? What is the hierarchy?
What does it mean for a queue to be blocked?
Does increasing queue size help?
How do queues being blocked relate to hardware on the machine?
Is there documentation anywhere on splunk.com where we can find all relevant information in one page?

Tags (2)
1 Solution

Stephen_Sorkin
Splunk Employee
Splunk Employee
  1. Queues become blocked when some (index-time) component in the system can't service data as fast as data is entering the system. For example, if date parsing can only operate at 30K E/s, and we could read data from files at >30K E/s, all upstream queues from date parsing will block.
  2. The standard index time queues and processors are: [inputs] -> parsingQueue -> [utf8 processor, line breaker, header parsing] -> aggQueue -> [date parsing and line merging] -> typingQueue -> [regex replacement, punct:: addition] -> indexQueue -> [tcp output, syslog output, http output, block signing, indexing, indexing metrics].
  3. If a queue is blocked, that means that data can't be added into the queue and the upstream pipeline will wait until it's empty.
  4. No, in steady state changing a queue's size will not materially impact when it gets blocked or system throughput characteristics. Blockage is simply caused by a mismatch of input to output rate.

View solution in original post

bong0004
Engager

Some questions are answered here:

http://wiki.splunk.com/Community:HowIndexingWorks

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee
  1. Queues become blocked when some (index-time) component in the system can't service data as fast as data is entering the system. For example, if date parsing can only operate at 30K E/s, and we could read data from files at >30K E/s, all upstream queues from date parsing will block.
  2. The standard index time queues and processors are: [inputs] -> parsingQueue -> [utf8 processor, line breaker, header parsing] -> aggQueue -> [date parsing and line merging] -> typingQueue -> [regex replacement, punct:: addition] -> indexQueue -> [tcp output, syslog output, http output, block signing, indexing, indexing metrics].
  3. If a queue is blocked, that means that data can't be added into the queue and the upstream pipeline will wait until it's empty.
  4. No, in steady state changing a queue's size will not materially impact when it gets blocked or system throughput characteristics. Blockage is simply caused by a mismatch of input to output rate.

hugocvg
Explorer

what is the recommendable relation between queues size?

0 Karma

rroberts
Splunk Employee
Splunk Employee

Is event typing also handled by the typingQueue?

0 Karma

Genti
Splunk Employee
Splunk Employee

The idea is that i would like an answers page where one can go and get most "general" questions regarding queues in one place, instead of checking multiple answers pages to get the big picture. I know that a doc is being worked on answering these questions and when completed hopefully a link to this page will be given.

0 Karma

Simeon
Splunk Employee
Splunk Employee

Thats a lot of questions at once... you should break out your questions or have a more direct one.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...