Alerting

Deployed Saved Searches Not Triggering Shell Scripts

mallem
Path Finder

I pushed multiple saved searches from the deployment head to many production deployment clients. On the clients, I can run the deployed saved searches from the "Searches & Reports" drop-down under the Search UI; the searches all show valid Scheduled times under Manager » Searches and reports (although they show "No Owner" under the Owner column); and my search and scheduler activity from the "Status" drop-down under the Search UI looks good. Yet, they do not trigger any scripts, even echo.sh (under $SPLUNK_HOME/bin/scripts). This is an issue as the script trigger handles sending saved search matches to our monitoring console. Any help is appreciated. I'm dead in the water with this functionality not working.

Tags (2)

mallem
Path Finder

We were using a v3.x command to execute the shell scripts (e.g. action_script = echo.sh). Once replaced by v4.x commands (e.g. action.script = 1 and action.script.filename = echo.sh), everything started working consistently. The "action_script" command worked intermittently, which threw me off. I would have suspected it would not have worked at all.

bbingham
Builder

I'd attach on, what OS is your splunk instance running on? I'm assuming *nix starting the echo.sh script, does the splunk user have a shell defined? Does the splunk user have rights (rwx or r_x) all the way to the directory and to the script you're trying to start?

If you happen to be running on OSX, how did you start splunk? OSX 10.6.2 and higher has a security patch only allowing scripts to be started if the master shell is still open, meaning if you started splunk using an ssh connection, once you exit out, splunk will continue to run but splunk can not start child processes. If this seems like your problem, you'll need to start splunk using launchd.

Agreeing with jrodman, there are a lot of variables to look at and you should probably open a support case to get the functionality going.

0 Karma

jrodman
Splunk Employee
Splunk Employee

This really needs to be a support case with http://www.splunk.com/support

If this is setting up the functionality for the first time, I would recommend reviewing your alert conditions to see if they are getting met.

If this is something which previously worked and has now stopped, I would recommend reviewing what has changed recently in your environment as well as calling into the support line.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...