Deployment Architecture

Maximum Number of Concurrent Running Jobs Search Head Cluster Error

ykpramodhcbt
Path Finder

Hi,

We have a search head cluster with 3 search heads -

We are seeing the following message and a decent skip ratio on DMC.

priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide", concurrency_limit=1

This might be the way it is designed to run searches - only in one search head at a given time. If it turns out to be a bug, please mention how we can fix this.

regards
Pramodh

1 Solution

baldwintm
Path Finder

Do these events have a "savedsearch_name" in them? how often is that search scheduled to run?
I think that's the message that indicates that a saved search is trying to run, but a previous instance of that search is still running (splunk will only allow one instance of a savedsearch to run at a time).

for example: if a saved search is scheduled to run every minute, but it takes five minutes to run, then it will skip the next four times that it tries to run.

View solution in original post

baldwintm
Path Finder

Do these events have a "savedsearch_name" in them? how often is that search scheduled to run?
I think that's the message that indicates that a saved search is trying to run, but a previous instance of that search is still running (splunk will only allow one instance of a savedsearch to run at a time).

for example: if a saved search is scheduled to run every minute, but it takes five minutes to run, then it will skip the next four times that it tries to run.

ykpramodhcbt
Path Finder

We observed this behavior when searches are skipped.

0 Karma

gayappanjan
New Member

Hi ,
Thankyou for the info
could you please provide solution for this type of issues?

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

HI @ykpramodhcbt,

Can you please check this answer?

https://answers.splunk.com/answers/337992/maxsearches-limit-reached.html

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...