Skipped search - The maximum number of concurrent running jobs for this historical scheduled search on this cluster has

[mchoudhary]

Hi Team,

I have been getting a skipped search notification in my CMC overview under Health from quite some time.
It is a scheduled report
Search name: ESS - Notable Events
Cron: every 5 mins ( 1-59/5 * * * *)
Timerange:  earliest - 48d@d  ; latest - +0s (now)
Message: The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached
Search query: 

`notable` 
| search NOT `suppression` 
| eval timeDiff_type=case(_time>=relative_time(now(), "-24h@h"),"current", 1=1, "historical") 
| expandtoken rule_title | table _time,event_id,security_domain,urgency,rule_name,rule_title,src,dest,src_user,user,dvc,status,status_group,owner,timeDiff_type,governance,control 
| outputlookup es_notable_events 
| stats count

It is writing the output to an output-lookup.  and takes around 8 mins as runtime when checked under job management.

Can some help me understand where the issue lies, what's making this search in particular to skip. The percentage skipped it around 50% and the status is critical.

[livehybrid]

Hi @mchoudhary 

The reason for the warning you are seeing is because its scheduled to run more frequently than the duration it takes to run. e.g. 5 minutes into the 8 minute runtime it wants to be running again.

There are two approaches to take here, which vary depending on your setup and stack load. Basically you want that search to take less than 5 minutes to run. Given that its an ES out of the box search it shouldnt need modifying (and doing so could end up with incorrect results). One thing I would want you to check though is to check what the `notable` macro expands to. If you find the macro through the UI (Settings -> Advanced Search -> Search macros) can you see the definition for the "notable" macro? Has this been modified at all?

The other thing that could be impacting the time it takes for the search to run is the load on the service. What does your architecture look like? Is your deployment based on the recommended reference hardware spec? (Min 16 Physical cores and 32GB RAM for the Searchheads) Check out https://help.splunk.com/en/splunk-enterprise-security-8/install/8.1/planning/minimum-specifications-... for more information.

Are there any other long-running searches which could be contributing to the load on the stack and thus slowing this search down? 

In other words, its going to take some digging, sometimes the search that takes longer to run isnt the problematic search - its just starved of resource and cant run faster!

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing