Alerting

Deployed Saved Searches Not Triggering Shell Scripts

mallem
Path Finder

I pushed multiple saved searches from the deployment head to many production deployment clients. On the clients, I can run the deployed saved searches from the "Searches & Reports" drop-down under the Search UI; the searches all show valid Scheduled times under Manager » Searches and reports (although they show "No Owner" under the Owner column); and my search and scheduler activity from the "Status" drop-down under the Search UI looks good. Yet, they do not trigger any scripts, even echo.sh (under $SPLUNK_HOME/bin/scripts). This is an issue as the script trigger handles sending saved search matches to our monitoring console. Any help is appreciated. I'm dead in the water with this functionality not working.

Tags (2)

mallem
Path Finder

We were using a v3.x command to execute the shell scripts (e.g. action_script = echo.sh). Once replaced by v4.x commands (e.g. action.script = 1 and action.script.filename = echo.sh), everything started working consistently. The "action_script" command worked intermittently, which threw me off. I would have suspected it would not have worked at all.

bbingham
Builder

I'd attach on, what OS is your splunk instance running on? I'm assuming *nix starting the echo.sh script, does the splunk user have a shell defined? Does the splunk user have rights (rwx or r_x) all the way to the directory and to the script you're trying to start?

If you happen to be running on OSX, how did you start splunk? OSX 10.6.2 and higher has a security patch only allowing scripts to be started if the master shell is still open, meaning if you started splunk using an ssh connection, once you exit out, splunk will continue to run but splunk can not start child processes. If this seems like your problem, you'll need to start splunk using launchd.

Agreeing with jrodman, there are a lot of variables to look at and you should probably open a support case to get the functionality going.

0 Karma

jrodman
Splunk Employee
Splunk Employee

This really needs to be a support case with http://www.splunk.com/support

If this is setting up the functionality for the first time, I would recommend reviewing your alert conditions to see if they are getting met.

If this is something which previously worked and has now stopped, I would recommend reviewing what has changed recently in your environment as well as calling into the support line.

0 Karma
Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...