Monitoring Splunk

Deployment monitor causing "maximum historical concurrent searches" warning?

Super Champion

I started seeing the following message at the top of the Splunk Web page after installing the Deployment Monitor app.

The maximum number of historical concurrent system-wide searches has been reached. current=36 maximum=36

Obviously, the current value varies depending on the load, but I seem to run into a very high number of scheduled searches especially at the top of every hour.

I did a little more investigating, specifically looking at the savedsearches.conf file shipped with this app and found that the following counts for scheduled intervals:

$ cat savedsearches.conf | grep cron_schedule | sed -re 's/^(.*)[ ]+$/\1/' | sort | uniq -c
  2 cron_schedule = 0 * * * *
  5 cron_schedule = 0 1 * * *
  3 cron_schedule = */15 * * * *
 20 cron_schedule = */30 * * * *

So this means that at the top of the hour 25 saved searches are scheduled to run. (Unless it's 1 AM, then there will be 30). Every half hour there will be 23 searches running, and 3 searches will run at 15 and 45 after.

So what's the recommendation here? Is it expected that the splunk admin of each site modify all these schedules to fit their environment? Or should I be tweaking my limits.conf settings to just hide the issue?

I just recently spent many hours shifting around the times of my custom scheduled searches to avoid this problem. Then I enabled this app, and now I'm back to square one.


I wouldn't modify limits.conf to hide this, as that might cause performance issues. I have the Deployment Server App only running on my Deployment server, so I don't see that many issues with it. I would suggest moving your Deploy Server and the App to a different host (if you can) and see if that helps. Other than that, modify the cron schedule to some off-beat times.

07,37 * * * *
03,18,33,48 * * * *

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...