topic Re: What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed" in Splunk Search

What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

SamHTexas — Thu, 25 Mar 2021 14:08:09 GMT

What do I need to check / do to resolve this please?

What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

Re: What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

richgalloway — Thu, 25 Mar 2021 19:18:27 GMT

Searches are delayed when there are no resources available at run-time and they have a non-zero Schedule Window. The delay lasts until the schedule window closes. If, at that time, the search still can't run then it becomes "skipped".

To resolve it, re-schedule the searches so fewer are scheduled at the same time. Pay particular attention to the :00, :15, :30, and :45 minutes of each hour. See https://github.com/dpaper-splunk/public/blob/master/dashboards/extended_search_reporting.xml for a helpful dashboard.

Re: What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

SamHTexas — Tue, 30 Mar 2021 01:06:18 GMT

Please tell me how to use the resource you listed o github. Thanks very much.

Re: What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

isoutamo — Tue, 30 Mar 2021 06:14:34 GMT

Just copy paste it to your node where you have those delayed searches as a dashboard.
Another option is use MC's Search -> Scheduler and look there what those searches are.
Anyhow you should look that time by time or create alert to inform you if there are lot of skipped or delayed searches.
r. Ismo

Re: What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

SamHTexas — Tue, 30 Mar 2021 19:10:10 GMT

Thank u for your message. I went to Monitoring console - Search - Scheduler Activity - Instance. All I see are "Search is waiting for input" in different windows. Please advise. Thx

Re: What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

richgalloway — Tue, 30 Mar 2021 21:03:28 GMT

Make sure each dropdown has something in it. Verify the MC is running in distributed mode and that each search head is a search peer to the MC.

Re: What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

isoutamo — Wed, 31 Mar 2021 05:00:17 GMT

You could found more information about MC from https://docs.splunk.com/Documentation/Splunk/8.1.3/DMC/DMCoverview

Re: What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

SamHTexas — Thu, 01 Apr 2021 20:05:52 GMT

Sir, what is the out come of using the github search you shared on a SH in Splunk. It ran for a while but no reports or messages appeared. Please advise. Thank you in advance.

What causes delayed searches alerts in Splunk Enterprise - Error says "searches delayed"

SamHTexas — Thu, 01 Apr 2021 20:10:36 GMT

Sir, what is the out come of using the github search you shared on a SH in Splunk. It ran for a while but no reports or messages appeared. Please advise. Thank you in advance.