Splunk Enterprise

Resource issues with indexing and queues

d_lim
Path Finder

Hi all, so I am facing this issue with what seems to be delayed/not receiving logs from the UFs.

d_lim_1-1613984409525.png

This is the current queue, we have gone from 1, to 2, to now 3 for the indexer's parallelIngestionPipeline settings.

On the indexer, the index queue is always full and is affecting the downstream from the 2 HFs. 

There are about 16 intermediate forwarders sending to HF001, and HF002 is mainly doing API calls to pull data.

The iops for the indexer is around 1600, cpu usage 50% and memory 31%.

Any recommendations on what we can do to improve this, eg. additional indexer? Thanks.

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

I full indexing queue means the act of writing to disk is taking too long.  Adding pipelines just makes that worse by creating more threads that try to write to disk.  Something in the storage system is causing delays and correcting that problem should alleviate the queue problem.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

I full indexing queue means the act of writing to disk is taking too long.  Adding pipelines just makes that worse by creating more threads that try to write to disk.  Something in the storage system is causing delays and correcting that problem should alleviate the queue problem.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...