topic Re: Splunk searches delayed in Monitoring Splunk

Splunk searches delayed

AliMaher — Mon, 24 Feb 2025 19:26:37 GMT

Hello,

I faced the below ERROR:

The percentage of non high priority searches delayed (27%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=18. Total delayed Searches=5

Search for the result:

Re: Splunk searches delayed

richgalloway — Tue, 25 Feb 2025 01:27:01 GMT

You're trying to run too many searches. There are several things you can do about that.

Convert real-time searches into scheduled searches.
Disable the searches you don't need.
Run the remaining searches less often.
Re-schedule searches so they are evenly distributed around the clock. Use the Extended Search Reporting dashboard to find the busy and not-so-busy times.
Add more CPUs to your search head(s). You also may need to add CPUs to the indexers. Do this only after performing all of the above tasks.

Re: Splunk searches delayed

AliMaher — Tue, 25 Feb 2025 09:25:27 GMT

Thanks for your help, really appreciated!

As per the below Screenshot:

Convert real-time searches into scheduled searches.
is real time = Ad-hoc?

Could you please assist in differentiate the difference between the (Historical - Realtime - Summarization - Ad-hoc) Searches?

Re: Splunk searches delayed

richgalloway — Wed, 26 Feb 2025 02:30:44 GMT

A real-time search runs continuously so matching events are returned as soon as they reach the indexer (before writing to disk).

Ad-hoc searches can be real-time, but they are not equivalent. "ad-hoc" refers to any non-scheduled search.

Historical searches look back in time for matching events.

Summarization searches aggregate results into a summary index for later processing.

The screenshot shows the system to be nearly idle and must have been taken when the health indicator was green (or is about to turn green - it can take up to 24 hours for the health indicator to reset).