Deployment Architecture

Maximum Number of Concurrent Running Jobs Search Head Cluster Error

ykpramodhcbt
Path Finder

Hi,

We have a search head cluster with 3 search heads -

We are seeing the following message and a decent skip ratio on DMC.

priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide", concurrency_limit=1

This might be the way it is designed to run searches - only in one search head at a given time. If it turns out to be a bug, please mention how we can fix this.

regards
Pramodh

1 Solution

baldwintm
Path Finder

Do these events have a "savedsearch_name" in them? how often is that search scheduled to run?
I think that's the message that indicates that a saved search is trying to run, but a previous instance of that search is still running (splunk will only allow one instance of a savedsearch to run at a time).

for example: if a saved search is scheduled to run every minute, but it takes five minutes to run, then it will skip the next four times that it tries to run.

View solution in original post

baldwintm
Path Finder

Do these events have a "savedsearch_name" in them? how often is that search scheduled to run?
I think that's the message that indicates that a saved search is trying to run, but a previous instance of that search is still running (splunk will only allow one instance of a savedsearch to run at a time).

for example: if a saved search is scheduled to run every minute, but it takes five minutes to run, then it will skip the next four times that it tries to run.

ykpramodhcbt
Path Finder

We observed this behavior when searches are skipped.

0 Karma

gayappanjan
New Member

Hi ,
Thankyou for the info
could you please provide solution for this type of issues?

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

HI @ykpramodhcbt,

Can you please check this answer?

https://answers.splunk.com/answers/337992/maxsearches-limit-reached.html

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...

Data Persistence in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. What happens if the OpenTelemetry collector ...

Thanks for the Memories! Splunk University, .conf25, and our Community

Thank you to everyone in the Splunk Community who joined us for .conf25, which kicked off with our iconic ...