Splunk Enterprise

Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment?

SamHTexas
Builder

Why would I get "searches Delayed" under health check on one SH not other SHs in a clustered environment? Shouldn't all Search heads have the same issue of " delayed searches"? 

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

As I understand it, each SH checks it's own health - not that of other members of the SHC.  Some SHs may have delayed searches because their searches are taking longer to run so newly-scheduled ones have to wait.

---
If this reply helps you, Karma would be appreciated.
0 Karma

SamHTexas
Builder

Thank u for your reply. I still am getting "searches delayed" when running health status. But am not getting far with using the monitoring console to find causes. Yes the MC is in Distributed mode & all the indexers are showing as peers. Under MC - search- Dist. search- instance. Still getting " waiting for input in windows presented with red exclamation marks. Please advise.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure all of the search heads are also peers to the MC.

A good dashboard that can help with search issues is at https://github.com/dpaper-splunk/public/blob/master/dashboards/extended_search_reporting.xml

---
If this reply helps you, Karma would be appreciated.

SamHTexas
Builder

I copied the script from github & pasted it in a SH waited for a long time. Did not receive any response or errors. Is this Daashbd. supposed to alert you when delayed are encountered? I know we have many delayed searches reported my the Monitoring console. 

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The dashboard does not alert.  It offers insight into the scheduling of searches (among other things) so you can see when you have too many searches trying to run at once.  Spread them out and the delays should go away.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...