Splunk Enterprise

Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment?

SamHTexas
Builder

Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment? Shouldn't all Search heads have the same issue of " delayed searches"? 

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

As I understand it, each SH checks it's own health - not that of other members of the SHC.  Some SHs may have delayed searches because their searches are taking longer to run so newly-scheduled ones have to wait.

---
If this reply helps you, Karma would be appreciated.
0 Karma

SamHTexas
Builder

Thank u for your reply. I still am getting "searches delayed" when running health status. But am not getting far with using the monitoring console to find causes. Yes the MC is in Distributed mode & all the indexers are showing as peers. Under MC - search- Dist. search- instance. Still getting " waiting for input in windows presented with red exclamation marks. Please advise.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure all of the search heads are also peers to the MC.

A good dashboard that can help with search issues is at https://github.com/dpaper-splunk/public/blob/master/dashboards/extended_search_reporting.xml

---
If this reply helps you, Karma would be appreciated.

SamHTexas
Builder

I copied the script from github & pasted it in a SH waited for a long time. Did not receive any response or errors. Is this Daashbd. supposed to alert you when delayed are encountered? I know we have many delayed searches reported my the Monitoring console. 

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The dashboard does not alert.  It offers insight into the scheduling of searches (among other things) so you can see when you have too many searches trying to run at once.  Spread them out and the delays should go away.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...