Why dose UF have parsingQueue and how to control the size?

[brandy81]

Hi, I have a question for UF.

1. From the capture below, it seems that UF has parsingQueue. As I understand, UF dose not parse. Parsing is HF or Indexer's role. Am I wrong? Why is there parsingQueue inside UF pipeline? (Let's say I just collect log data, not structured-csv file.)

2. If it is correct that UF has parsingQueue, how to control the size? Is it related to maxQueueSize in outputs.conf or [queue] in limits.conf?

3. From below image, what is difference between parsingQueue and tcpout_queue, and how to control size for each of them?

[Shetry]

Hello, there

I hope you are doing well.

I was studying Splunk basics and came to an image that made me ask the same question you have asked here, but I don't understand the explanation.

I would be grateful if you could explain to my why the UF has a parsing queue in it 

Thank you 

[tscroggins]

Hi @Shetry,

This should be posted as a new question, but briefly, Splunk Universal Forwarder and Splunk Enterprise share the same (or a similar) codebase. Binary detection, event breaking, and more are handled in parsingQueue. If force_local_processing is enabled in props.conf, line breaking, timestamp extraction, and transforms can also be handled by a universal forwarder.

See the following for a high resolution PDF of the last (v7.2) pipeline diagram. It's still applicable today, but you'll need to cross reference Splunk documentation for the latest features.

https://web.archive.org/web/20220125091543/https://wiki.splunk.com/Community:HowIndexingWorks

https://web.archive.org/web/20220125091543/https://wiki.splunk.com/File:Splunk_EventProcessing_v20_1...

[Shetry]

Thank you so much for your help!
Much appreciated

[gjanders]

FYI the corrected wording should be "Why does UF..."

If you refer to server.conf.spec

[queue=<queueName>]

In this case the keyword is parsingQueue and you can adjust the size

outputs.conf.spec

The max queue size is the TCP output size...it is controlled per-output queue

In this case parsing would be reading the data off disk, in the splexicon parsingQueue

"A queue in the data pipeline that holds data after it enters the system, but before parsing (event processing) occurs.

Incoming data goes first to the parsingQueue and from there to the parsing pipeline, where it undergoes event processing. It then moves to the indexQueue and on to the indexing pipeline, which builds the index."

[brandy81]

@gjanders  Thanks a lot. It helped a lot.

In the diagram, there is a parsing pipeline on universal forwarder. Do you mean the parsing pipeline on universal forwarder do not do "event processing"? If yes, why does it name "parsing pipeline"?  It makes me confused.

[gjanders · Accepted solution]

@brandy81I agree with the confusing terminology! The parsing pipeline on a UF exists (I think) but it is different to the parsing pipeline on an enterprise instance.

In the community wiki diagram which is not an official Splunk doc, if you refer to the other diagram on https://wiki.splunk.com/Community:HowIndexingWorks , at the time of writing https://wiki.splunk.com/File:Splunk_EventProcessing_v20.0_Standalone.png you can see the parsing queue is definitely different...(e..g the line breaker)