Splunk Enterprise

Why dose UF have parsingQueue and how to control the size?

brandy81
Path Finder

Hi, I have a question for UF.

 

1. From the capture below, it seems that UF has parsingQueue. As I understand, UF dose not parse. Parsing is HF or Indexer's role. Am I wrong? Why is there parsingQueue inside UF pipeline? (Let's say I just collect log data, not structured-csv file.)

2. If it is correct that UF has parsingQueue, how to control the size? Is it related to maxQueueSize in outputs.conf or [queue] in limits.conf?

3. From below image, what is difference between parsingQueue and tcpout_queue, and how to control size for each of them?

Screen Shot 2020-07-16 at 10.47.44 AM.png

Labels (1)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

@brandy81I agree with the confusing terminology! The parsing pipeline on a UF exists (I think) but it is different to the parsing pipeline on an enterprise instance.

In the community wiki diagram which is not an official Splunk doc, if you refer to the other diagram on https://wiki.splunk.com/Community:HowIndexingWorks , at the time of writing https://wiki.splunk.com/File:Splunk_EventProcessing_v20.0_Standalone.png you can see the parsing queue is definitely different...(e..g the line breaker)

View solution in original post

gjanders
SplunkTrust
SplunkTrust

FYI the corrected wording should be "Why does UF..."

If you refer to server.conf.spec

[queue=<queueName>]

In this case the keyword is parsingQueue and you can adjust the size

outputs.conf.spec

The max queue size is the TCP output size...it is controlled per-output queue

In this case parsing would be reading the data off disk, in the splexicon parsingQueue

"A queue in the data pipeline that holds data after it enters the system, but before parsing (event processing) occurs.

Incoming data goes first to the parsingQueue and from there to the parsing pipeline, where it undergoes event processing. It then moves to the indexQueue and on to the indexing pipeline, which builds the index."

0 Karma

brandy81
Path Finder

@gjanders  Thanks a lot. It helped a lot.

In the diagram, there is a parsing pipeline on universal forwarder. Do you mean the parsing pipeline on universal forwarder do not do "event processing"? If yes, why does it name "parsing pipeline"?  It makes me confused.

0 Karma

gjanders
SplunkTrust
SplunkTrust

@brandy81I agree with the confusing terminology! The parsing pipeline on a UF exists (I think) but it is different to the parsing pipeline on an enterprise instance.

In the community wiki diagram which is not an official Splunk doc, if you refer to the other diagram on https://wiki.splunk.com/Community:HowIndexingWorks , at the time of writing https://wiki.splunk.com/File:Splunk_EventProcessing_v20.0_Standalone.png you can see the parsing queue is definitely different...(e..g the line breaker)

*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>