Deployment monitor causing "maximum historical concurrent searches" warning?

[Lowell]

I started seeing the following message at the top of the Splunk Web page after installing the Deployment Monitor app.

The maximum number of historical concurrent system-wide searches has been reached. current=36 maximum=36

Obviously, the current value varies depending on the load, but I seem to run into a very high number of scheduled searches especially at the top of every hour.

I did a little more investigating, specifically looking at the savedsearches.conf file shipped with this app and found that the following counts for scheduled intervals:

$ cat savedsearches.conf | grep cron_schedule | sed -re 's/^(.*)[ ]+$/\1/' | sort | uniq -c
  2 cron_schedule = 0 * * * *
  5 cron_schedule = 0 1 * * *
  3 cron_schedule = */15 * * * *
 20 cron_schedule = */30 * * * *

So this means that at the top of the hour 25 saved searches are scheduled to run. (Unless it's 1 AM, then there will be 30). Every half hour there will be 23 searches running, and 3 searches will run at 15 and 45 after.

So what's the recommendation here? Is it expected that the splunk admin of each site modify all these schedules to fit their environment? Or should I be tweaking my limits.conf settings to just hide the issue?

I just recently spent many hours shifting around the times of my custom scheduled searches to avoid this problem. Then I enabled this app, and now I'm back to square one.

[alacercogitatus]

I wouldn't modify limits.conf to hide this, as that might cause performance issues. I have the Deployment Server App only running on my Deployment server, so I don't see that many issues with it. I would suggest moving your Deploy Server and the App to a different host (if you can) and see if that helps. Other than that, modify the cron schedule to some off-beat times.

07,37 * * * *
03,18,33,48 * * * *