maxsearches limit reached

[kamlesh_vaghela]

Hi ,
I have configured 20 Alerts with below run every - "5 min" and Alert mode "Once Per Result". Each Alerts are generating more then 1000 events . As the Alert Action , a shell script is called.
So with this , Some of the alerts are executed properly and others are skipped. See below log from "scheduler.log".

12-23-2015 17:44:58.494 +0530 INFO SavedSplunker - savedsearch_id="admin;search;Alert 20", user="admin", app="search", savedsearch_name="Alert 20", status=skipped, reason="maxsearches limit reached", scheduled_time=1450872600, window_time=0

Kindly suggest me what should I do in this kind of situations.

Thanks

[jkat54 · Accepted solution]

Please read the upvoted answer on this question:

https://answers.splunk.com/answers/297030/what-is-the-max-value-for-maxsearches-is-there-a-w.html

Short answer: this can be adjusted in limits.conf.
Warning: limits.conf must be on peers and search heads for this setting to work in distributed environment

Long answer: why not stagger your searches so that you're not executing 20 at the exact same time.?

*/5 * * * *   <- every five on the hour
1-59/5 * * * * <- every five starting at 1 minute after the hour
2-59/5 * * * * <- every five starting at 2 minutes after the hour

[peter_krammer]

Which limit in limits.conf do I need to change to allow the search to be queued?
We have alerts that we definitely do not want to be skipped, but sometimes they take longer to execute and the following search is skipped because the first is not finished yet.

[biec1]

The following is the current configuration in our environment. We are getting max_concurrent limit reached on one of the three Search Heads for specific Saved Search. It does not happen on rest of the two Search Heads.
Search Head:- 3x16+10=58
Search Peer :- 1x20+6=26

Is it necessary to maintain identical parameters on both Search peers and Search heads?
Shall i increase max_searches_per_cpu on Search Peers from 1 to 2?