topic The percentage of non high priority searches delayed in Deployment Architecture

The percentage of non high priority searches delayed

bsanjeeva — Wed, 12 Jan 2022 00:20:16 GMT

Hi Splunkers,

I am getting below error on Clustered Search heard,

"The percentage of non high priority searches delayed (88%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=1615. Total delayed Searches=1430"

This issue is particularly seen on one app whose index receives large amount of data. How do I fix this issue?

Thanks in advance

Re: The percentage of non high priority searches delayed

richgalloway — Wed, 12 Jan 2022 01:12:07 GMT

Check the Scheduler Activity page of the Monitoring Console to try to determine why searches are delayed. The most likely cause is too many searches trying to run at the same time. Another likely cause is searches not completing before the next scheduled run-time.

To fix the first cause, reschedule searches so they're spread out across the clock. It's very common for most searches to try to run at :00 so focus on those first.

To fix the second cause you can make the search more efficient (faster) or reschedule it so it has time to complete before the next run-time.

Re: The percentage of non high priority searches delayed

bsanjeeva — Wed, 12 Jan 2022 15:39:28 GMT

@richgalloway , thanks for the suggestions. Here is my observations,

In MC's Scheduler Activity, I see that it issue is present only on one of the three search heads in cluster. Many Reports are skipped and deferred with following reason,

"The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached (4)"

Do I have to modify limits.conf ?

Why is this issue seen only one one search head in the cluster?

Thanks

Re: The percentage of non high priority searches delayed

richgalloway — Wed, 12 Jan 2022 16:29:12 GMT

That is a symptom of the second cause of skipped searches. The message is saying it is time to run a search, but an earlier invocation of the search is still running.

The fix is either 1) make the search complete sooner; or 2) schedule it to run less frequently.

It only happens on one SH because scheduled searches only run on one search head.

Re: The percentage of non high priority searches delayed

bsanjeeva — Wed, 12 Jan 2022 17:07:52 GMT

Now I understand the issue with your explanation. Can modifying limits.conf help in anyway?

Current config in searchhead:

[search]
dispatch_dir_warning_size=7000

Thanks

Re: The percentage of non high priority searches delayed

richgalloway — Wed, 12 Jan 2022 19:04:34 GMT

Changing limits.conf won't help in this case. You just can't have more than one instance of the same scheduled search running at the same time.