Splunk Search

What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

SamHTexas
Builder

What do I need to check / do to resolve this please?

What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Searches are delayed when there are no resources available at run-time and they have a non-zero Schedule Window.  The delay lasts until the schedule window closes.  If, at that time, the search still can't run then it becomes "skipped".

To resolve it, re-schedule the searches so fewer are scheduled at the same time.  Pay particular attention to the :00, :15, :30, and :45 minutes of each hour.  See https://github.com/dpaper-splunk/public/blob/master/dashboards/extended_search_reporting.xml for a helpful dashboard.

---
If this reply helps you, Karma would be appreciated.
0 Karma

SamHTexas
Builder

Please tell me how to use the resource you listed o github. Thanks very much.

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
Just copy paste it to your node where you have those delayed searches as a dashboard.
Another option is use MC's Search -> Scheduler and look there what those searches are.
Anyhow you should look that time by time or create alert to inform you if there are lot of skipped or delayed searches.
r. Ismo
0 Karma

SamHTexas
Builder

Sir, what is the out come of using the github search you shared on a SH in Splunk. It ran for a while but no reports or messages appeared. Please advise. Thank you in advance.

Tags (1)
0 Karma

SamHTexas
Builder

Sir, what is the out come of using the github search you shared on a SH in Splunk. It ran for a while but no reports or messages appeared. Please advise. Thank you in advance.

Tags (1)
0 Karma

SamHTexas
Builder

Thank u for your message. I went to Monitoring console - Search - Scheduler Activity - Instance. All I see are "Search is waiting for input" in different windows. Please advise. Thx

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure each dropdown has something in it.  Verify the MC is running in distributed mode and that each search head is a search peer to the MC.

---
If this reply helps you, Karma would be appreciated.
0 Karma

isoutamo
SplunkTrust
SplunkTrust
You could found more information about MC from https://docs.splunk.com/Documentation/Splunk/8.1.3/DMC/DMCoverview
0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...