Why dose UF have parsingQueue and how to control the size?

[brandy81]

Hi, I have a question for UF.

1. From the capture below, it seems that UF has parsingQueue. As I understand, UF dose not parse. Parsing is HF or Indexer's role. Am I wrong? Why is there parsingQueue inside UF pipeline? (Let's say I just collect log data, not structured-csv file.)

2. If it is correct that UF has parsingQueue, how to control the size? Is it related to maxQueueSize in outputs.conf or [queue] in limits.conf?

3. From below image, what is difference between parsingQueue and tcpout_queue, and how to control size for each of them?

[gjanders]

FYI the corrected wording should be "Why does UF..."

If you refer to server.conf.spec

[queue=<queueName>]

In this case the keyword is parsingQueue and you can adjust the size

outputs.conf.spec

The max queue size is the TCP output size...it is controlled per-output queue

In this case parsing would be reading the data off disk, in the splexicon parsingQueue

"A queue in the data pipeline that holds data after it enters the system, but before parsing (event processing) occurs.

Incoming data goes first to the parsingQueue and from there to the parsing pipeline, where it undergoes event processing. It then moves to the indexQueue and on to the indexing pipeline, which builds the index."

[brandy81]

@gjanders  Thanks a lot. It helped a lot.

In the diagram, there is a parsing pipeline on universal forwarder. Do you mean the parsing pipeline on universal forwarder do not do "event processing"? If yes, why does it name "parsing pipeline"?  It makes me confused.

[gjanders · Accepted solution]

@brandy81I agree with the confusing terminology! The parsing pipeline on a UF exists (I think) but it is different to the parsing pipeline on an enterprise instance.

In the community wiki diagram which is not an official Splunk doc, if you refer to the other diagram on https://wiki.splunk.com/Community:HowIndexingWorks , at the time of writing https://wiki.splunk.com/File:Splunk_EventProcessing_v20.0_Standalone.png you can see the parsing queue is definitely different...(e..g the line breaker)