Saturated Event-Processing Queues

[msplunk33]

I am getting this error frequently and I can see the index queue is 99% for many indexers in the cluster. I am not able to figure out what is causing this issue. During this period indexing is considerable slow and logs are not ingesting for many source type. I am not able to figure out what is causing this issue(which source). After sometime it go back to normal. I am worried this can case issue in the future.

[richgalloway]

In the MC, select Indexing->Indexing Performance: Instance.  Then scroll down to the "Estimated Indexing Rate Per Sourcetype" panel.  Use the dropdown menu to split the graph by various attributes until you find the source of the problem.

[richgalloway]

A full queue is caused by a slow-down after the queue or a sudden increase before the queue.

Check your storage system to make sure there is nothing that is causing the I/O rate to drop significantly, like an AV scan.  Splunk should not be sharing storage with other high-I/O applications like a DB.

A periodic surge in incoming data can also lead to backed-up queues.  Use the monitoring console to see what sources contributed a lot of data during the period of the slowdown.

[msplunk33]

@richgalloway 

Use the monitoring console to see what sources contributed a lot of data during the period of the slowdown.

I could not find the above option in the monitoring console. Could you give me the menu details  from the monitoring console or a scereenshot.