Splunk Enterprise

Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment?

SamHTexas
Contributor

Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment? Shouldn't all Search heads have the same issue of " delayed searches"? 

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

As I understand it, each SH checks it's own health - not that of other members of the SHC.  Some SHs may have delayed searches because their searches are taking longer to run so newly-scheduled ones have to wait.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

SamHTexas
Contributor

Thank u for your reply. I still am getting "searches delayed" when running health status. But am not getting far with using the monitoring console to find causes. Yes the MC is in Distributed mode & all the indexers are showing as peers. Under MC - search- Dist. search- instance. Still getting " waiting for input in windows presented with red exclamation marks. Please advise.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure all of the search heads are also peers to the MC.

A good dashboard that can help with search issues is at https://github.com/dpaper-splunk/public/blob/master/dashboards/extended_search_reporting.xml

---
If this reply helps you, an upvote would be appreciated.

SamHTexas
Contributor

I copied the script from github & pasted it in a SH waited for a long time. Did not receive any response or errors. Is this Daashbd. supposed to alert you when delayed are encountered? I know we have many delayed searches reported my the Monitoring console. 

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The dashboard does not alert.  It offers insight into the scheduling of searches (among other things) so you can see when you have too many searches trying to run at once.  Spread them out and the delays should go away.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!