topic Re: Saturated Event-Processing Queues in Monitoring Splunk

Saturated Event-Processing Queues

msplunk33 — Wed, 21 Oct 2020 00:59:47 GMT

I am getting this error frequently and I can see the index queue is 99% for many indexers in the cluster. I am not able to figure out what is causing this issue. During this period indexing is considerable slow and logs are not ingesting for many source type. I am not able to figure out what is causing this issue(which source). After sometime it go back to normal. I am worried this can case issue in the future.

Re: Saturated Event-Processing Queues

richgalloway — Wed, 21 Oct 2020 13:25:15 GMT

A full queue is caused by a slow-down after the queue or a sudden increase before the queue.

Check your storage system to make sure there is nothing that is causing the I/O rate to drop significantly, like an AV scan. Splunk should not be sharing storage with other high-I/O applications like a DB.

A periodic surge in incoming data can also lead to backed-up queues. Use the monitoring console to see what sources contributed a lot of data during the period of the slowdown.

Re: Saturated Event-Processing Queues

msplunk33 — Wed, 21 Oct 2020 23:46:48 GMT

@richgalloway

Use the monitoring console to see what sources contributed a lot of data during the period of the slowdown.

I could not find the above option in the monitoring console. Could you give me the menu details from the monitoring console or a scereenshot.

Re: Saturated Event-Processing Queues

richgalloway — Thu, 22 Oct 2020 13:49:25 GMT

In the MC, select Indexing->Indexing Performance: Instance. Then scroll down to the "Estimated Indexing Rate Per Sourcetype" panel. Use the dropdown menu to split the graph by various attributes until you find the source of the problem.