Getting Data In

Parsing queue and Aggregation queue sizes on heavy forwarders?

jihape
Path Finder

We are seeing the aggregation and parsing queues almost constantly flatlining at a 100% on our HFs. On our indexers the queues are at 0% pretty much all the time, with the occasional spike to 20%.

Would increasing the [queue=parsingQueue] and [queue=aggQueue] in system/local/server.conf help lowering the queue fill ratio? The current settings are the default 6MB for Parsing and 1MB for aggregation.

There are only 250 UFs connected to the HF, but that is expected to increase dramatically.

If increasing the queue size works, by how much should I increase it? Is there a rule of thumb that I haven't seen?

Tags (1)
0 Karma

sohaib112
Explorer

Facing this same issue with few servers.
@jihape Have you found the solution of it? How did you fix it?

I have just added the this setting `parallelIngestionPipelines=2` and increased the parsingqueue on HF from 1024MB to 2048MB. Would this solve the issue? 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

how many HFs you have vs indexers? I have seen a rule of thumb that you should have (min) 7 hf pipeline per one IDX pipeline to get this working smoothly.

r. Ismo

0 Karma

justynap_ldz
Path Finder

Hi @jihape,  have you solved this issue? We are facing the same problem...

0 Karma

nareshinsvu
Builder

@jihape - Was your issue resolved? I am in the same boat looking for a safe shore.

0 Karma

HiroshiSatoh
Champion

The load on the indexer is low and the queue is formed on the HF side is a problem in the processing on the HF side.
Increasing the size of the queue does not solve the problem. Please first check high load processing.
For example, the transfer rate (maxKBps) may be low.Also, if the parsing process is slow, it is also possible to increase the pipeline.

0 Karma

jihape
Path Finder

Thanks, I couldn't identify any issues with the server:
1. It was not a CPU issue - max CPU was only 50%
2. transfer rate in limits.conf is set to 0
3. Is it this settings you are talking about when saying 'increase the pipeline'? http://docs.splunk.com/Documentation/Splunk/7.0.2/Capacity/Parallelization#Index_parallelization

0 Karma

HiroshiSatoh
Champion

Yes. The pipeline is that. Processes can be parallelized. But up to two. More than (3 or more require PS.

Can you investigate high load queues?
setting>monitoring console>Indexing>Indexing Performance: Instance

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...