Splunk Search

What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

SamHTexas
Builder

What do I need to check / do to resolve this please?

What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Searches are delayed when there are no resources available at run-time and they have a non-zero Schedule Window.  The delay lasts until the schedule window closes.  If, at that time, the search still can't run then it becomes "skipped".

To resolve it, re-schedule the searches so fewer are scheduled at the same time.  Pay particular attention to the :00, :15, :30, and :45 minutes of each hour.  See https://github.com/dpaper-splunk/public/blob/master/dashboards/extended_search_reporting.xml for a helpful dashboard.

---
If this reply helps you, Karma would be appreciated.
0 Karma

SamHTexas
Builder

Please tell me how to use the resource you listed o github. Thanks very much.

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
Just copy paste it to your node where you have those delayed searches as a dashboard.
Another option is use MC's Search -> Scheduler and look there what those searches are.
Anyhow you should look that time by time or create alert to inform you if there are lot of skipped or delayed searches.
r. Ismo
0 Karma

SamHTexas
Builder

Sir, what is the out come of using the github search you shared on a SH in Splunk. It ran for a while but no reports or messages appeared. Please advise. Thank you in advance.

Tags (1)
0 Karma

SamHTexas
Builder

Sir, what is the out come of using the github search you shared on a SH in Splunk. It ran for a while but no reports or messages appeared. Please advise. Thank you in advance.

Tags (1)
0 Karma

SamHTexas
Builder

Thank u for your message. I went to Monitoring console - Search - Scheduler Activity - Instance. All I see are "Search is waiting for input" in different windows. Please advise. Thx

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure each dropdown has something in it.  Verify the MC is running in distributed mode and that each search head is a search peer to the MC.

---
If this reply helps you, Karma would be appreciated.
0 Karma

isoutamo
SplunkTrust
SplunkTrust
You could found more information about MC from https://docs.splunk.com/Documentation/Splunk/8.1.3/DMC/DMCoverview
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...