Splunk Enterprise

Resource issues with indexing and queues

d_lim
Path Finder

Hi all, so I am facing this issue with what seems to be delayed/not receiving logs from the UFs.

d_lim_1-1613984409525.png

This is the current queue, we have gone from 1, to 2, to now 3 for the indexer's parallelIngestionPipeline settings.

On the indexer, the index queue is always full and is affecting the downstream from the 2 HFs. 

There are about 16 intermediate forwarders sending to HF001, and HF002 is mainly doing API calls to pull data.

The iops for the indexer is around 1600, cpu usage 50% and memory 31%.

Any recommendations on what we can do to improve this, eg. additional indexer? Thanks.

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

I full indexing queue means the act of writing to disk is taking too long.  Adding pipelines just makes that worse by creating more threads that try to write to disk.  Something in the storage system is causing delays and correcting that problem should alleviate the queue problem.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

I full indexing queue means the act of writing to disk is taking too long.  Adding pipelines just makes that worse by creating more threads that try to write to disk.  Something in the storage system is causing delays and correcting that problem should alleviate the queue problem.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...