The percentage of high priority searches skipped

[scqing]

Hello, My splunk cluster have a alert like" The percentage of high priority searches skipped (21%) over the last 24 hours is very high and exceeded the red thresholds (10%) on this Splunk instance. Total Searches that were part of this percentage=23. Total skipped Searches=5 The percentage of non high priority searches skipped (22%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=8835. Total skipped Searches=1947"。 What can I do，The Splunk stopped work now!! I have tried to Change the running time of rules and  to disperse them as much as possible,But still not work. Thanks, Jason

[richgalloway]

Searches are skipped when there are no resources available to run them at the scheduled time.  There are a few ways to address that:

1) Re-schedule the searches so fewer try to run at the same time.

2) Improve the performance of searches so they complete sooner.

3) Run heavy-weight searches during off hours so they're not competing with ad-hoc searches (which have priority).

4) Increase the number of searches per CPU (if the CPUs are not too busy)

5) Add more CPUs to the search head

6) Add more SHs to the SHC (or create a SHC if you don't have one)

Please explain what you mean by "Splunk stopped work".  I've never seen skipped searches stop Splunk before.

[scqing]

"Splunk stopped work" means all rules stopped working，until restart the splunk SH.

1) Re-schedule the searches so fewer try to run at the same time.

I tried already。

2) Improve the performance of searches so they complete sooner.

3) Run heavy-weight searches during off hours so they're not competing with ad-hoc searches (which have priority).

4) Increase the number of searches per CPU (if the CPUs are not too busy)

According to monitoring，I don't think CPU is busy。

5) Add more CPUs to the search head

6) Add more SHs to the SHC (or create a SHC if you don't have one) 

For the same monitoring rules, my old environment is a stand-alone version of Splunk, version 6.5.1. Now the same host configuration is expanded to three hosts and clustered. Version 8.1.2 .however， performance is bad, so it's not easy to apply for resources again

[isoutamo]

Hi

You probably have MC configured on your environment. It's not matter if you have distributed or standalone environment. In distributed environment this needs some additional steps to set up.

On both environments (MC node or your standalone node) open Settings -> Monitoring Console -> Search -> Scheduler Activity: Deployment/Instance (depend on your environment).

Check which instance has those skipped searches then select for it Instance specific dashboard.

On that Dashboard there are several items which told different views for skipped/deferred searches. By those you will get the understanding why those have skipped. After that it should relative easy to figure out what are correct actions from @richgalloway 's list.

r. Ismo