Alerting

Deployed Saved Searches Not Triggering Shell Scripts

mallem
Path Finder

I pushed multiple saved searches from the deployment head to many production deployment clients. On the clients, I can run the deployed saved searches from the "Searches & Reports" drop-down under the Search UI; the searches all show valid Scheduled times under Manager » Searches and reports (although they show "No Owner" under the Owner column); and my search and scheduler activity from the "Status" drop-down under the Search UI looks good. Yet, they do not trigger any scripts, even echo.sh (under $SPLUNK_HOME/bin/scripts). This is an issue as the script trigger handles sending saved search matches to our monitoring console. Any help is appreciated. I'm dead in the water with this functionality not working.

Tags (2)

mallem
Path Finder

We were using a v3.x command to execute the shell scripts (e.g. action_script = echo.sh). Once replaced by v4.x commands (e.g. action.script = 1 and action.script.filename = echo.sh), everything started working consistently. The "action_script" command worked intermittently, which threw me off. I would have suspected it would not have worked at all.

bbingham
Builder

I'd attach on, what OS is your splunk instance running on? I'm assuming *nix starting the echo.sh script, does the splunk user have a shell defined? Does the splunk user have rights (rwx or r_x) all the way to the directory and to the script you're trying to start?

If you happen to be running on OSX, how did you start splunk? OSX 10.6.2 and higher has a security patch only allowing scripts to be started if the master shell is still open, meaning if you started splunk using an ssh connection, once you exit out, splunk will continue to run but splunk can not start child processes. If this seems like your problem, you'll need to start splunk using launchd.

Agreeing with jrodman, there are a lot of variables to look at and you should probably open a support case to get the functionality going.

0 Karma

jrodman
Splunk Employee
Splunk Employee

This really needs to be a support case with http://www.splunk.com/support

If this is setting up the functionality for the first time, I would recommend reviewing your alert conditions to see if they are getting met.

If this is something which previously worked and has now stopped, I would recommend reviewing what has changed recently in your environment as well as calling into the support line.

0 Karma
Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...