Skipped search error on CMC | The maximum number of concurrent running jobs.....on this cluster has been reached

[mchoudhary]

Hi Team,

I have been observing 1 skipped search error indicating on my CMC. Error is -
"The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached"

Percentage skipped is 33.3%. 

I saw many solutions online, stating to increase the user/role search job limit or either make changes in the limits.conf (which I don't have access to) but couldn't figure out or get clear explanations, Can someone help me to solve this.

Also, the saved search in question is running on a cron of 2-59/5 * * * * for time range of 5 mins. 

Please suggest.

[richgalloway]

The message is generated when the scheduler sees it's time to run a search, but a previous instance of the same search is still running.  Usually, that happens because the search is trying to do too much can't complete the query before the next run interval.

You have a few options:

1. Make the search more efficient so it runs faster and completes sooner
2. Increase the schedule interval
3. Run the search at a less-busy time
4. Re-schedule other resource-intensive searches so they don't compete with the one getting skipped.

[livehybrid]

Hi @mchoudhary 

This typically happens when a scheduled search takes longer to complete than the interval between its scheduled runs, or when multiple demanding searches are scheduled to run at the same time, exhausting the available concurrency slots.

The cron schedule 2-59/5 * * * * means your search attempts to run every 5 minutes (at 2, 7, 12, ... minutes past the hour). The 33.3% skip rate (1 out of 3 runs skipped) strongly suggests that this search takes longer than 10 minutes but less than 15 minutes to complete, and the concurrency limit for these types of searches under your user/role context is effectively 2. This pattern would cause two instances of the search to start and run concurrently, and the third attempt would be skipped because both available slots are still occupied.

Before changing any limits to the user/role etc I would investigate how long the search is running for, and if possible improve the performance of the search. 

Please can you check how long it is taking to run? If you are able to post the offending search here we may be able to help improve its performance.

Try the following search to get the duration of your scheduled searches:

index=_audit search_id="'scheduler*" info=completed | stats avg(total_run_time) as avg_total_run_time, p95(total_run_time) as p95_total_run_time by savedsearch_name

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing