Deployment Architecture

Maximum Number of Concurrent Running Jobs Search Head Cluster Error

ykpramodhcbt
Path Finder

Hi,

We have a search head cluster with 3 search heads -

We are seeing the following message and a decent skip ratio on DMC.

priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide", concurrency_limit=1

This might be the way it is designed to run searches - only in one search head at a given time. If it turns out to be a bug, please mention how we can fix this.

regards
Pramodh

1 Solution

baldwintm
Path Finder

Do these events have a "savedsearch_name" in them? how often is that search scheduled to run?
I think that's the message that indicates that a saved search is trying to run, but a previous instance of that search is still running (splunk will only allow one instance of a savedsearch to run at a time).

for example: if a saved search is scheduled to run every minute, but it takes five minutes to run, then it will skip the next four times that it tries to run.

View solution in original post

baldwintm
Path Finder

Do these events have a "savedsearch_name" in them? how often is that search scheduled to run?
I think that's the message that indicates that a saved search is trying to run, but a previous instance of that search is still running (splunk will only allow one instance of a savedsearch to run at a time).

for example: if a saved search is scheduled to run every minute, but it takes five minutes to run, then it will skip the next four times that it tries to run.

ykpramodhcbt
Path Finder

We observed this behavior when searches are skipped.

0 Karma

gayappanjan
New Member

Hi ,
Thankyou for the info
could you please provide solution for this type of issues?

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

HI @ykpramodhcbt,

Can you please check this answer?

https://answers.splunk.com/answers/337992/maxsearches-limit-reached.html

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...