Splunk Enterprise

Resource issues with indexing and queues

d_lim
Path Finder

Hi all, so I am facing this issue with what seems to be delayed/not receiving logs from the UFs.

d_lim_1-1613984409525.png

This is the current queue, we have gone from 1, to 2, to now 3 for the indexer's parallelIngestionPipeline settings.

On the indexer, the index queue is always full and is affecting the downstream from the 2 HFs. 

There are about 16 intermediate forwarders sending to HF001, and HF002 is mainly doing API calls to pull data.

The iops for the indexer is around 1600, cpu usage 50% and memory 31%.

Any recommendations on what we can do to improve this, eg. additional indexer? Thanks.

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

I full indexing queue means the act of writing to disk is taking too long.  Adding pipelines just makes that worse by creating more threads that try to write to disk.  Something in the storage system is causing delays and correcting that problem should alleviate the queue problem.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

I full indexing queue means the act of writing to disk is taking too long.  Adding pipelines just makes that worse by creating more threads that try to write to disk.  Something in the storage system is causing delays and correcting that problem should alleviate the queue problem.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...