topic Re: Is there a time limit on saved search scripts? in Alerting

Is there a time limit on saved search scripts?

heybigben — Tue, 09 Jun 2020 21:51:14 GMT

I've got a splunk saved search configured to run an external script when number of events > 0. Are there any limits on how long the external script can take to run before being killed by splunk? The script simply processes the search results and updates a external datastore. The script can take between 1 seconds and 6 minutes to run, depending upon the number of events and how busy the server is. I'm seeing the following messages ocasionally in the splunkd.log

01-02-2012 02:20:33.824 -0800 WARN ScriptRunner - Killing script, probably timed out, grace=5sec, script="/local/mnt/workspace/splunk/etc/apps/search/bin/runshellscript.py"

01-02-2012 02:20:33.827 -0800 ERROR script - Script execution failed for external search command 'runshellscript'

Is there really a grace limit of 5 seconds? If so, any way to increase the timeout and allow the script to complete?

Re: Is there a time limit on saved search scripts?

hexx — Tue, 10 Jan 2012 05:25:55 GMT

Looking at savedsearches.conf, I do not see any parameters that would allow to configure a timeout period for a scripted alert. As it is, it seems that unfortunately the script execution timeout period is hard-coded at 5 minutes with a grace period of 5 seconds.

If you would like this to be configurable, I would suggest that you submit an enhancement request by opening a support case explaining your use-case and the desired added functionality.

Re: Is there a time limit on saved search scripts?

Damien_Dallimor — Tue, 10 Jan 2012 06:27:12 GMT

In lieu of being able to configure the timeout value for the fired script , you could perhaps create a solution using the Splunk REST API.

ie:

write a python script that is cron triggered.
python script uses the Splunk Python SDK to execute your search
then proceed with your existing logic if search returns "events > 0"

Re: Is there a time limit on saved search scripts?

nembela — Wed, 17 Apr 2019 13:29:50 GMT

I don't know if this problem relevant yet but I had the same problem. Splunk support found an undocumented paramater for this in savedseaches.conf:

action.script.maxtime
The maximum amount of time a script action takes before the action is canceled. The valid format is number followed by a time unit ("s", "m", "h", or "d").

I hope it helps.

Re: Is there a time limit on saved search scripts?

chandanjaisal — Wed, 30 Sep 2020 05:16:29 GMT

https://answers.splunk.com/answers/653458/max-time-spent-in-per-result-alerts-issue.html

limits.conf

[scheduler]
action_execution_threads = 10
actions_queue_size = 10000
max_per_result_alerts =10000
max_per_result_alerts_time = 600