topic Re: Parsing queue and Aggregation queue sizes on heavy forwarders? in Getting Data In

Parsing queue and Aggregation queue sizes on heavy forwarders?

jihape — Thu, 01 Mar 2018 00:15:50 GMT

We are seeing the aggregation and parsing queues almost constantly flatlining at a 100% on our HFs. On our indexers the queues are at 0% pretty much all the time, with the occasional spike to 20%.

Would increasing the [queue=parsingQueue] and [queue=aggQueue] in system/local/server.conf help lowering the queue fill ratio? The current settings are the default 6MB for Parsing and 1MB for aggregation.

There are only 250 UFs connected to the HF, but that is expected to increase dramatically.

If increasing the queue size works, by how much should I increase it? Is there a rule of thumb that I haven't seen?

Re: Parsing queue and Aggregation queue sizes on heavy forwarders?

HiroshiSatoh — Thu, 01 Mar 2018 03:17:54 GMT

The load on the indexer is low and the queue is formed on the HF side is a problem in the processing on the HF side.
Increasing the size of the queue does not solve the problem. Please first check high load processing.
For example, the transfer rate (maxKBps) may be low.Also, if the parsing process is slow, it is also possible to increase the pipeline.

Re: Parsing queue and Aggregation queue sizes on heavy forwarders?

jihape — Thu, 01 Mar 2018 03:56:12 GMT

Thanks, I couldn't identify any issues with the server:
1. It was not a CPU issue - max CPU was only 50%
2. transfer rate in limits.conf is set to 0
3. Is it this settings you are talking about when saying 'increase the pipeline'? http://docs.splunk.com/Documentation/Splunk/7.0.2/Capacity/Parallelization#Index_parallelization

Re: Parsing queue and Aggregation queue sizes on heavy forwarders?

HiroshiSatoh — Thu, 01 Mar 2018 05:13:04 GMT

Yes. The pipeline is that. Processes can be parallelized. But up to two. More than (3 or more require PS.

Can you investigate high load queues?
setting>monitoring console>Indexing>Indexing Performance: Instance

Re: Parsing queue and Aggregation queue sizes on heavy forwarders?

nareshinsvu — Mon, 26 Aug 2019 01:16:00 GMT

@jihape - Was your issue resolved? I am in the same boat looking for a safe shore.

Re: Parsing queue and Aggregation queue sizes on heavy forwarders?

justynap_ldz — Wed, 06 Jul 2022 15:23:38 GMT

Hi @jihape, have you solved this issue? We are facing the same problem...

Re: Parsing queue and Aggregation queue sizes on heavy forwarders?

sohaib112 — Fri, 20 Jan 2023 16:00:31 GMT

Facing this same issue with few servers.
@jihape Have you found the solution of it? How did you fix it?

I have just added the this setting `parallelIngestionPipelines=2` and increased the parsingqueue on HF from 1024MB to 2048MB. Would this solve the issue?

Re: Parsing queue and Aggregation queue sizes on heavy forwarders?

isoutamo — Fri, 20 Jan 2023 21:48:59 GMT

how many HFs you have vs indexers? I have seen a rule of thumb that you should have (min) 7 hf pipeline per one IDX pipeline to get this working smoothly.

r. Ismo