topic The percentage of high priority searches skipped in Splunk Enterprise

The percentage of high priority searches skipped

scqing — Tue, 02 Nov 2021 09:53:19 GMT

Hello, My splunk cluster have a alert like" The percentage of high priority searches skipped (21%) over the last 24 hours is very high and exceeded the red thresholds (10%) on this Splunk instance. Total Searches that were part of this percentage=23. Total skipped Searches=5 The percentage of non high priority searches skipped (22%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=8835. Total skipped Searches=1947"。 What can I do，The Splunk stopped work now!! I have tried to Change the running time of rules and to disperse them as much as possible,But still not work. Thanks, Jason

Re: The percentage of high priority searches skipped

richgalloway — Tue, 02 Nov 2021 19:25:04 GMT

Searches are skipped when there are no resources available to run them at the scheduled time. There are a few ways to address that:

1) Re-schedule the searches so fewer try to run at the same time.

2) Improve the performance of searches so they complete sooner.

3) Run heavy-weight searches during off hours so they're not competing with ad-hoc searches (which have priority).

4) Increase the number of searches per CPU (if the CPUs are not too busy)

5) Add more CPUs to the search head

6) Add more SHs to the SHC (or create a SHC if you don't have one)

Please explain what you mean by "Splunk stopped work". I've never seen skipped searches stop Splunk before.

Re: The percentage of high priority searches skipped

scqing — Tue, 11 Jan 2022 07:00:48 GMT

"Splunk stopped work" means all rules stopped working，until restart the splunk SH.

1) Re-schedule the searches so fewer try to run at the same time.

I tried already。

2) Improve the performance of searches so they complete sooner.

3) Run heavy-weight searches during off hours so they're not competing with ad-hoc searches (which have priority).

4) Increase the number of searches per CPU (if the CPUs are not too busy)

According to monitoring，I don't think CPU is busy。

5) Add more CPUs to the search head

6) Add more SHs to the SHC (or create a SHC if you don't have one)

For the same monitoring rules, my old environment is a stand-alone version of Splunk, version 6.5.1. Now the same host configuration is expanded to three hosts and clustered. Version 8.1.2 .however， performance is bad, so it's not easy to apply for resources again

Re: The percentage of high priority searches skipped

isoutamo — Tue, 11 Jan 2022 09:39:22 GMT

You probably have MC configured on your environment. It's not matter if you have distributed or standalone environment. In distributed environment this needs some additional steps to set up.

On both environments (MC node or your standalone node) open Settings -> Monitoring Console -> Search -> Scheduler Activity: Deployment/Instance (depend on your environment).

Check which instance has those skipped searches then select for it Instance specific dashboard.

On that Dashboard there are several items which told different views for skipped/deferred searches. By those you will get the understanding why those have skipped. After that it should relative easy to figure out what are correct actions from @richgalloway 's list.

r. Ismo