topic Re: Why dose UF have parsingQueue and how to control the size? in Splunk Enterprise

Why dose UF have parsingQueue and how to control the size?

brandy81 — Thu, 16 Jul 2020 01:57:31 GMT

Hi, I have a question for UF.

1. From the capture below, it seems that UF has parsingQueue. As I understand, UF dose not parse. Parsing is HF or Indexer's role. Am I wrong? Why is there parsingQueue inside UF pipeline? (Let's say I just collect log data, not structured-csv file.)

2. If it is correct that UF has parsingQueue, how to control the size? Is it related to maxQueueSize in outputs.conf or [queue] in limits.conf?

3. From below image, what is difference between parsingQueue and tcpout_queue, and how to control size for each of them?

Re: Why dose UF have parsingQueue and how to control the size?

gjanders — Tue, 21 Jul 2020 04:52:30 GMT

FYI the corrected wording should be "Why does UF..."

If you refer to server.conf.spec

[queue=<queueName>]

In this case the keyword is parsingQueue and you can adjust the size

outputs.conf.spec

The max queue size is the TCP output size...it is controlled per-output queue

In this case parsing would be reading the data off disk, in the splexicon parsingQueue

"A queue in the data pipeline that holds data after it enters the system, but before parsing (event processing) occurs.

Incoming data goes first to the parsingQueue and from there to the parsing pipeline, where it undergoes event processing. It then moves to the indexQueue and on to the indexing pipeline, which builds the index."

Re: Why dose UF have parsingQueue and how to control the size?

brandy81 — Tue, 21 Jul 2020 05:33:49 GMT

@gjanders Thanks a lot. It helped a lot.

In the diagram, there is a parsing pipeline on universal forwarder. Do you mean the parsing pipeline on universal forwarder do not do "event processing"? If yes, why does it name "parsing pipeline"? It makes me confused.

Re: Why dose UF have parsingQueue and how to control the size?

gjanders — Tue, 21 Jul 2020 06:27:05 GMT

@brandy81I agree with the confusing terminology! The parsing pipeline on a UF exists (I think) but it is different to the parsing pipeline on an enterprise instance.

In the community wiki diagram which is not an official Splunk doc, if you refer to the other diagram on https://wiki.splunk.com/Community:HowIndexingWorks , at the time of writing https://wiki.splunk.com/File:Splunk_EventProcessing_v20.0_Standalone.png you can see the parsing queue is definitely different...(e..g the line breaker)

Re: Why dose UF have parsingQueue and how to control the size?

Shetry — Sat, 22 Jun 2024 19:35:27 GMT

Hello, there

I hope you are doing well.

I was studying Splunk basics and came to an image that made me ask the same question you have asked here, but I don't understand the explanation.

I would be grateful if you could explain to my why the UF has a parsing queue in it

Thank you

Re: Why dose UF have parsingQueue and how to control the size?

tscroggins — Sun, 23 Jun 2024 15:55:23 GMT

Hi @Shetry,

This should be posted as a new question, but briefly, Splunk Universal Forwarder and Splunk Enterprise share the same (or a similar) codebase. Binary detection, event breaking, and more are handled in parsingQueue. If force_local_processing is enabled in props.conf, line breaking, timestamp extraction, and transforms can also be handled by a universal forwarder.

See the following for a high resolution PDF of the last (v7.2) pipeline diagram. It's still applicable today, but you'll need to cross reference Splunk documentation for the latest features.

https://web.archive.org/web/20220125091543/https://wiki.splunk.com/Community:HowIndexingWorks

https://web.archive.org/web/20220125091543/https://wiki.splunk.com/File:Splunk_EventProcessing_v20_1_UF_Indexer.pdf

Re: Why dose UF have parsingQueue and how to control the size?

Shetry — Sun, 23 Jun 2024 17:41:44 GMT

Thank you so much for your help!
Much appreciated