Parsing queue and Aggregation queue sizes on heavy forwarders?

[jihape]

We are seeing the aggregation and parsing queues almost constantly flatlining at a 100% on our HFs. On our indexers the queues are at 0% pretty much all the time, with the occasional spike to 20%.

Would increasing the [queue=parsingQueue] and [queue=aggQueue] in system/local/server.conf help lowering the queue fill ratio? The current settings are the default 6MB for Parsing and 1MB for aggregation.

There are only 250 UFs connected to the HF, but that is expected to increase dramatically.

If increasing the queue size works, by how much should I increase it? Is there a rule of thumb that I haven't seen?

[sohaib112]

Facing this same issue with few servers.
@jihape Have you found the solution of it? How did you fix it?

I have just added the this setting `parallelIngestionPipelines=2` and increased the parsingqueue on HF from 1024MB to 2048MB. Would this solve the issue? 

[isoutamo]

Hi

how many HFs you have vs indexers? I have seen a rule of thumb that you should have (min) 7 hf pipeline per one IDX pipeline to get this working smoothly.

r. Ismo

[justynap_ldz]

Hi @jihape,  have you solved this issue? We are facing the same problem...

[nareshinsvu]

@jihape - Was your issue resolved? I am in the same boat looking for a safe shore.

[HiroshiSatoh]

The load on the indexer is low and the queue is formed on the HF side is a problem in the processing on the HF side.
Increasing the size of the queue does not solve the problem. Please first check high load processing.
For example, the transfer rate (maxKBps) may be low.Also, if the parsing process is slow, it is also possible to increase the pipeline.

[jihape]

Thanks, I couldn't identify any issues with the server:
1. It was not a CPU issue - max CPU was only 50%
2. transfer rate in limits.conf is set to 0
3. Is it this settings you are talking about when saying 'increase the pipeline'? http://docs.splunk.com/Documentation/Splunk/7.0.2/Capacity/Parallelization#Index_parallelization

[HiroshiSatoh]

Yes. The pipeline is that. Processes can be parallelized. But up to two. More than (3 or more require PS.

Can you investigate high load queues?
setting>monitoring console>Indexing>Indexing Performance: Instance