Splunk Enterprise

Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment?

SamHTexas
Builder

Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment? Shouldn't all Search heads have the same issue of " delayed searches"? 

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

As I understand it, each SH checks it's own health - not that of other members of the SHC.  Some SHs may have delayed searches because their searches are taking longer to run so newly-scheduled ones have to wait.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

SamHTexas
Builder

Thank u for your reply. I still am getting "searches delayed" when running health status. But am not getting far with using the monitoring console to find causes. Yes the MC is in Distributed mode & all the indexers are showing as peers. Under MC - search- Dist. search- instance. Still getting " waiting for input in windows presented with red exclamation marks. Please advise.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure all of the search heads are also peers to the MC.

A good dashboard that can help with search issues is at https://github.com/dpaper-splunk/public/blob/master/dashboards/extended_search_reporting.xml

---
If this reply helps you, an upvote would be appreciated.

SamHTexas
Builder

I copied the script from github & pasted it in a SH waited for a long time. Did not receive any response or errors. Is this Daashbd. supposed to alert you when delayed are encountered? I know we have many delayed searches reported my the Monitoring console. 

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The dashboard does not alert.  It offers insight into the scheduling of searches (among other things) so you can see when you have too many searches trying to run at once.  Spread them out and the delays should go away.

---
If this reply helps you, an upvote would be appreciated.
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!