Skipped searches

[VijaySrrie]

Hi All,

I am working on skipped searches, what is the difference between below 2?

1) The maximum number of concurrent historical scheduled searches on this cluster has been reached

2) The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached

[Tom_Lundie · Accepted solution]

Hi,

The wording is quite tricky but I will do my best to explain:

1) The maximum number of concurrent historical scheduled searches on this cluster has been reached:

Platform-level. Essentially,  the Splunk platform has reached its concurrent search limit as defined by the concurrency settings in limits.conf. For example, if the limit is five, you might have six different searches all scheduled at once, triggering this error.

2) The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached

This is defined on a per-search level under max_concurrent within savedsearches.conf (advanced search settings). It is one by default. This means that one particular search has been scheduled whilst previous instances of that same search is still running. So for example, you schedule a particular search to run every two minutes but it takes six minutes to run. Splunk starts it, and two minutes later, goes to schedule it for a second time but its still running, throwing the error.

As a tip, 1) is typically caused by incorrectly scheduled searches all scheduled for the same time. Whereas 2) is caused by a particular search being scheduled too frequently or running for longer than expected, causing it to overlap with itself.