Skipped searches issue

[Helios]

We have standalone environment and are getting error "the percentage of non-high priority searches skipped (61%) over the last 24 hours is very high and exceeded the red threshold (20%) on this splunk instance." 

The environment:

Customer has standalone where we created an app with a savedsearch script that pulls all indexed events every 1 hour and bundles them into a  .json file, customer then compresses it into a .gz file for transfer into our production environment.  

What we are seeing is this skipped searches message and when we check the specific job, we see that every time it runs there are 2 things that come up as jobs, the export app started by python calling the script and then the actual search job activity with our SPL search, both jobs are 1 second apart and stays in the jobs page for 10 minutes each, customer states that it takes ~2.5 minutes for this job to complete.   The python script seems to stay longer for some reason, even after its job 

Not sure how to proceed, since we had it scheduled every 4 hours and it was doing the same thing, so we lowered it to 1 hour, no difference.

Our search looks at the last completed .json file epoch time and current epoch time to grab those events in that range, so not sure if that message is like a false positive by the way we are catching events (timestamps).  How can i remove the skipped searches error message.  Tips??

[Helios]

Okay Thank you

[gcusello]

Hi @Helios ,

let me know if I can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[Helios]

additional info.

We searched the error, and found that:

"The maximum number of concurrent running jobs for a historical scheduled search has been reached."

Now, we have export python script running, the error shows that is this python export script that is causing problems, with concurrent jobs maybe

[gcusello]

Hi @Helios,

the fist question is: what are the hardware resources you have on your server? Splunk requires at least 12 CPUs and 12 GB RAM, usually the issue is related to the CPUs.

this seems that you don't have sufficient resources (eventually only on some time periods) to run all the scheduled searches and many of them are skipped.

So analyze, using the Monitoring Console, the searches, to understand if there's a resource problem or you need only to define a different scheduling for the savedsearches execution.

Last check: how many real time searches have you in execution?

remember that a Splunk search uses a CPU for each search (more than 1 if you have subsearches) and release them only when the search is finished (never for real time searches!), so if you have two o three real time searches in execution, there's the risk to finish the resources.

In this case, schedule the execution of these searches using fixed time periods.

Ciao.

Giuseppe