Splunk Enterprise

pipeline paralllelization ingest_pipe=0 not processing data

aab5272
Engager

Hi ,

I changed the pipelineparalleization=2

and i was able to see the data being processed through both pipelines, but after increasing the typingqueue , my one of ingest_pipe is not processing the data?

Can some one please let me know why would this happen. It was working before.

Thanks

Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

You can use this search to look at your queues size per pipeline, and see where it gets full. ( we are using perc90 of the queue filled size)

  index=_internal host=* source=*metrics.log sourcetype=splunkd group=queue (name=parsingqueue OR name=aggqueue OR name=typingqueue OR name=indexqueue) | eval name=case(name=="aggqueue","2 - Aggregation Queue",name=="indexqueue","4 - Indexing Queue",name=="parsingqueue","1 - Parsing Queue",name=="typingqueue","3 - Typing Queue") | eval ingest_pipe = if(isnotnull(ingest_pipe), ingest_pipe, "none") | search ingest_pipe=* | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size)  | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size)  | eval fill_perc=round((curr/max)*100,2) | eval name=name."-".ingest_pipe | timechart  Perc90(fill_perc) by name useother=false limit=15

We will order the processor in the order they run, so that way you can see if one is full and is backing up the previous ones.
1- parsing -> 2 -aggregation -> 3 -typing -> 4- indexing

0 Karma
Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...