Monitoring Splunk

Saturated Event-Processing Queues

msplunk33
Path Finder

I am getting this error frequently and I can see the index queue is 99% for many indexers in the cluster. I am not able to figure out what is causing this issue. During this period indexing is considerable slow and logs are not ingesting for many source type. I am not able to figure out what is causing this issue(which source). After sometime it go back to normal. I am worried this can case issue in the future.

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

In the MC, select Indexing->Indexing Performance: Instance.  Then scroll down to the "Estimated Indexing Rate Per Sourcetype" panel.  Use the dropdown menu to split the graph by various attributes until you find the source of the problem.

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

A full queue is caused by a slow-down after the queue or a sudden increase before the queue.

Check your storage system to make sure there is nothing that is causing the I/O rate to drop significantly, like an AV scan.  Splunk should not be sharing storage with other high-I/O applications like a DB.

A periodic surge in incoming data can also lead to backed-up queues.  Use the monitoring console to see what sources contributed a lot of data during the period of the slowdown.

---
If this reply helps you, Karma would be appreciated.
0 Karma

msplunk33
Path Finder

@richgalloway 

 

Use the monitoring console to see what sources contributed a lot of data during the period of the slowdown.

 

I could not find the above option in the monitoring console. Could you give me the menu details  from the monitoring console or a scereenshot.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...