How can I solve for skipped searches?

[uagraw01]

Hello Splunkers !!

I have attached below two screenshot related to skip searches. As per the below graph many times we have high number of skip searches. When I validated those I seen that workload_pool are not assigned to many saved searched ( attached in second screenshot ).

My thought here :
Because If so many searches are triggering on the same time and there is no workload_pool setting assigned then it will impact in the search performance and increase the value of skip ratio.

Please let me know I am thinking on a right way ? If not please guide me or suggest me some good workarounds. I know there many blogs available on this. But please do share , if any specific suggestion on this.

[christhianb]

Hey @uagraw01 

There are different ways to fix it but everything depends on the reason of the skipped search. 

You can run index=_internal sourcetype=scheduler status=skipped | stats values(reason) by savedsearch_name

That should help you out.

Once you identity the reason, make decisions. i.e disable unnecessary alerts, reduce the Time range picker, improve the SPL. This could be a fix for the most common reason " Max Concurrent searches have been reached..." 

[gcusello]

Hi @uagraw01,

if you're using an on-premise installation, probably your hardware isn't sufficient to work all the scheduled searches you have.

Which reference hardware are you using? how many scheduled searches?

Ciao.

Giuseppe

[uagraw01]

@gcuselloIts Splunk Cloud, and there 40+ saved searches which are showing with no workload_pool

[gcusello]

Hi @uagraw01,

which kind of license are you using: indexed logs or SVC?

if SVC probably you are exceeding your license.

In this case ask to you Splunk partner.

Ciao.

Giuseppe

[uagraw01]

@gcuselloCan't we control with putting some new admission rule in workload management ?

[gcusello]

Hi @uagraw01,

you could reduce your scheduled searches,

Did you checked license and hardware?

Ciao.

Giuseppe