Splunk Search

maxsearches limit reached

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi ,
I have configured 20 Alerts with below run every - "5 min" and Alert mode "Once Per Result". Each Alerts are generating more then 1000 events . As the Alert Action , a shell script is called.
So with this , Some of the alerts are executed properly and others are skipped. See below log from "scheduler.log".

12-23-2015 17:44:58.494 +0530 INFO SavedSplunker - savedsearch_id="admin;search;Alert 20", user="admin", app="search", savedsearch_name="Alert 20", status=skipped, reason="maxsearches limit reached", scheduled_time=1450872600, window_time=0

Kindly suggest me what should I do in this kind of situations.

Thanks

Tags (1)
1 Solution

jkat54
SplunkTrust
SplunkTrust

Please read the upvoted answer on this question:

https://answers.splunk.com/answers/297030/what-is-the-max-value-for-maxsearches-is-there-a-w.html

Short answer: this can be adjusted in limits.conf.
Warning: limits.conf must be on peers and search heads for this setting to work in distributed environment

Long answer: why not stagger your searches so that you're not executing 20 at the exact same time.?

*/5 * * * *   <- every five on the hour
1-59/5 * * * * <- every five starting at 1 minute after the hour
2-59/5 * * * * <- every five starting at 2 minutes after the hour

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Please read the upvoted answer on this question:

https://answers.splunk.com/answers/297030/what-is-the-max-value-for-maxsearches-is-there-a-w.html

Short answer: this can be adjusted in limits.conf.
Warning: limits.conf must be on peers and search heads for this setting to work in distributed environment

Long answer: why not stagger your searches so that you're not executing 20 at the exact same time.?

*/5 * * * *   <- every five on the hour
1-59/5 * * * * <- every five starting at 1 minute after the hour
2-59/5 * * * * <- every five starting at 2 minutes after the hour

peter_krammer
Communicator

Which limit in limits.conf do I need to change to allow the search to be queued?
We have alerts that we definitely do not want to be skipped, but sometimes they take longer to execute and the following search is skipped because the first is not finished yet.

0 Karma

biec1
Explorer

The following is the current configuration in our environment. We are getting max_concurrent limit reached on one of the three Search Heads for specific Saved Search. It does not happen on rest of the two Search Heads.
Search Head:- 3x16+10=58
Search Peer :- 1x20+6=26

Is it necessary to maintain identical parameters on both Search peers and Search heads?
Shall i increase max_searches_per_cpu on Search Peers from 1 to 2?

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...

Secure Your Future: Mastering Upgrade Readiness for Splunk 10

Spotlight: The Splunk Health Assistant Add-On  The Splunk Health Assistant Add-On is your ultimate companion ...

Observability Unlocked: Kubernetes & Cloud Monitoring with Splunk IM

Ready to master Kubernetes and cloud monitoring like the pros? Join Splunk’s Growth Engineering team on ...