Maximum Number of Concurrent Running Jobs Search Head Cluster Error

[ykpramodhcbt]

Hi,

We have a search head cluster with 3 search heads -

We are seeing the following message and a decent skip ratio on DMC.

priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide", concurrency_limit=1

This might be the way it is designed to run searches - only in one search head at a given time. If it turns out to be a bug, please mention how we can fix this.

regards
Pramodh

[baldwintm · Accepted solution]

Do these events have a "savedsearch_name" in them? how often is that search scheduled to run?
I think that's the message that indicates that a saved search is trying to run, but a previous instance of that search is still running (splunk will only allow one instance of a savedsearch to run at a time).

for example: if a saved search is scheduled to run every minute, but it takes five minutes to run, then it will skip the next four times that it tries to run.

[ykpramodhcbt]

We observed this behavior when searches are skipped.

[gayappanjan]

Hi ,
Thankyou for the info
could you please provide solution for this type of issues?

[kamlesh_vaghela]

HI @ykpramodhcbt,

Can you please check this answer?

https://answers.splunk.com/answers/337992/maxsearches-limit-reached.html