Monitoring Splunk

Deployment monitor causing "maximum historical concurrent searches" warning?

Super Champion

I started seeing the following message at the top of the Splunk Web page after installing the Deployment Monitor app.

The maximum number of historical concurrent system-wide searches has been reached. current=36 maximum=36

Obviously, the current value varies depending on the load, but I seem to run into a very high number of scheduled searches especially at the top of every hour.

I did a little more investigating, specifically looking at the savedsearches.conf file shipped with this app and found that the following counts for scheduled intervals:

$ cat savedsearches.conf | grep cron_schedule | sed -re 's/^(.*)[ ]+$/\1/' | sort | uniq -c
  2 cron_schedule = 0 * * * *
  5 cron_schedule = 0 1 * * *
  3 cron_schedule = */15 * * * *
 20 cron_schedule = */30 * * * *

So this means that at the top of the hour 25 saved searches are scheduled to run. (Unless it's 1 AM, then there will be 30). Every half hour there will be 23 searches running, and 3 searches will run at 15 and 45 after.

So what's the recommendation here? Is it expected that the splunk admin of each site modify all these schedules to fit their environment? Or should I be tweaking my limits.conf settings to just hide the issue?

I just recently spent many hours shifting around the times of my custom scheduled searches to avoid this problem. Then I enabled this app, and now I'm back to square one.


I wouldn't modify limits.conf to hide this, as that might cause performance issues. I have the Deployment Server App only running on my Deployment server, so I don't see that many issues with it. I would suggest moving your Deploy Server and the App to a different host (if you can) and see if that helps. Other than that, modify the cron schedule to some off-beat times.

07,37 * * * *
03,18,33,48 * * * *

0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...