Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment?

[SamHTexas]

Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment? Shouldn't all Search heads have the same issue of " delayed searches"? 

[richgalloway]

As I understand it, each SH checks it's own health - not that of other members of the SHC.  Some SHs may have delayed searches because their searches are taking longer to run so newly-scheduled ones have to wait.

[SamHTexas]

Thank u for your reply. I still am getting "searches delayed" when running health status. But am not getting far with using the monitoring console to find causes. Yes the MC is in Distributed mode & all the indexers are showing as peers. Under MC - search- Dist. search- instance. Still getting " waiting for input in windows presented with red exclamation marks. Please advise.

[richgalloway]

Make sure all of the search heads are also peers to the MC.

A good dashboard that can help with search issues is at https://github.com/dpaper-splunk/public/blob/master/dashboards/extended_search_reporting.xml

[SamHTexas]

I copied the script from github & pasted it in a SH waited for a long time. Did not receive any response or errors. Is this Daashbd. supposed to alert you when delayed are encountered? I know we have many delayed searches reported my the Monitoring console. 

[richgalloway]

The dashboard does not alert.  It offers insight into the scheduling of searches (among other things) so you can see when you have too many searches trying to run at once.  Spread them out and the delays should go away.