Getting Data In

Parsing queue and Aggregation queue sizes on heavy forwarders?

jihape
Path Finder

We are seeing the aggregation and parsing queues almost constantly flatlining at a 100% on our HFs. On our indexers the queues are at 0% pretty much all the time, with the occasional spike to 20%.

Would increasing the [queue=parsingQueue] and [queue=aggQueue] in system/local/server.conf help lowering the queue fill ratio? The current settings are the default 6MB for Parsing and 1MB for aggregation.

There are only 250 UFs connected to the HF, but that is expected to increase dramatically.

If increasing the queue size works, by how much should I increase it? Is there a rule of thumb that I haven't seen?

Tags (1)
0 Karma

nareshinsvu
Builder

@jihape - Was your issue resolved? I am in the same boat looking for a safe shore.

0 Karma

HiroshiSatoh
Champion

The load on the indexer is low and the queue is formed on the HF side is a problem in the processing on the HF side.
Increasing the size of the queue does not solve the problem. Please first check high load processing.
For example, the transfer rate (maxKBps) may be low.Also, if the parsing process is slow, it is also possible to increase the pipeline.

0 Karma

jihape
Path Finder

Thanks, I couldn't identify any issues with the server:
1. It was not a CPU issue - max CPU was only 50%
2. transfer rate in limits.conf is set to 0
3. Is it this settings you are talking about when saying 'increase the pipeline'? http://docs.splunk.com/Documentation/Splunk/7.0.2/Capacity/Parallelization#Index_parallelization

0 Karma

HiroshiSatoh
Champion

Yes. The pipeline is that. Processes can be parallelized. But up to two. More than (3 or more require PS.

Can you investigate high load queues?
setting>monitoring console>Indexing>Indexing Performance: Instance

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.