Deployed Saved Searches Not Triggering Shell Scripts

[mallem]

I pushed multiple saved searches from the deployment head to many production deployment clients. On the clients, I can run the deployed saved searches from the "Searches & Reports" drop-down under the Search UI; the searches all show valid Scheduled times under Manager » Searches and reports (although they show "No Owner" under the Owner column); and my search and scheduler activity from the "Status" drop-down under the Search UI looks good. Yet, they do not trigger any scripts, even echo.sh (under $SPLUNK_HOME/bin/scripts). This is an issue as the script trigger handles sending saved search matches to our monitoring console. Any help is appreciated. I'm dead in the water with this functionality not working.

[mallem]

We were using a v3.x command to execute the shell scripts (e.g. action_script = echo.sh). Once replaced by v4.x commands (e.g. action.script = 1 and action.script.filename = echo.sh), everything started working consistently. The "action_script" command worked intermittently, which threw me off. I would have suspected it would not have worked at all.

[bbingham]

I'd attach on, what OS is your splunk instance running on? I'm assuming *nix starting the echo.sh script, does the splunk user have a shell defined? Does the splunk user have rights (rwx or r_x) all the way to the directory and to the script you're trying to start?

If you happen to be running on OSX, how did you start splunk? OSX 10.6.2 and higher has a security patch only allowing scripts to be started if the master shell is still open, meaning if you started splunk using an ssh connection, once you exit out, splunk will continue to run but splunk can not start child processes. If this seems like your problem, you'll need to start splunk using launchd.

Agreeing with jrodman, there are a lot of variables to look at and you should probably open a support case to get the functionality going.

[jrodman]

This really needs to be a support case with http://www.splunk.com/support

If this is setting up the functionality for the first time, I would recommend reviewing your alert conditions to see if they are getting met.

If this is something which previously worked and has now stopped, I would recommend reviewing what has changed recently in your environment as well as calling into the support line.