topic Re: Launching tower-cli playbook from Splunk alert in Alerting

Launching tower-cli playbook from Splunk alert

ndavis4 — Tue, 29 Sep 2020 20:07:38 GMT

I have setup an alert to run a tower-cli script that kicks off a playbook. I installed tower-cli in /opt/rh on my Splunk server. In order to troubleshoot, I have stripped the script down to just list some users and pipe it to a file. I can run the script at the cli but it will not run from the alert in Splunk. I know the script is kicking off as I also just have a echo command in the script the dumps the timestamp to the same file. Here is my shell script I am calling from my Splunk alert:

Location of script file is: /home/splunk/bin/scripts
nac_interfaces.sh

!/bin/bash

echo "date '+%m-%e-%G %H:%M:%S'" >> "/home/splunk/bin/scripts/nac_interfaces.log"
source /opt/rh/python27/enable
/opt/rh/python27/root/usr/bin/tower-cli user list >> "/home/splunk/bin/scripts/nac_interfaces.log"

This is an error I am getting when the above script tries to execute the nac_interfaces.sh via the alerts script in Splunk
ERROR script - sid:rt_scheduler_adminsearch_EUCImaging_at_1529626004_20372.268 command="runshellscript", Script: /opt/splunk/bin/scripts/nac_interfaces.sh exited with status code: 127

Does it matter that splunk is installed in /home with a symlink to /opt?
[root@splunk opt]# pwd
/opt
[root@splunk opt]# ls -ll
lrwxrwxrwx. 1 splunk splunk 12 Nov 19 2013 splunk -> /home/splunk

Re: Launching tower-cli playbook from Splunk alert

renjith_nair — Sat, 23 Jun 2018 09:58:49 GMT

Hi @ndavis4,

BASH throws the error code 127 when it's not able to find the executable in PATH system variable and it is not a built-in shell command. In other words, the system doesn't understand your command, because it doesn't know where to find the binary you're trying to call.
Try giving absolute path for all the executable and make sure that they have +x permissions and splunk user has permissions to execute them

Re: Launching tower-cli playbook from Splunk alert

ndavis4 — Mon, 25 Jun 2018 14:22:08 GMT

I do have the full path on my Splunk server to run the tower-cli script. I can run the actual tower-cli command that is in the alerts scripts from any path on the CLI.... "/opt/rh/python27/root/usr/bin/tower-cli user list >> "/home/splunk/bin/scripts/nac_interfaces.log"
Just can't execute it from the alerts scripts in Splunk

Not sure what path it can't find. The tower-cli is attempting to login to another server to run the playbook.

Here is the content of the tower-cli file:

[root@splunk bin]# cat tower-cli

#!/opt/rh/python27/root/usr/bin/python2

# -*- coding: utf-8 -*-
import re
import sys

from tower_cli.cli.run import cli

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())

Here is the tower-cli config. foobar would be the host that has the playbook to run

[root@splunk bin]# tower-cli config

# User options (set with `tower-cli config`; stored in ~/.tower_cli.cfg).
host: foobar.com
username: splunk.service
password: xxxxxxxxx
verify_ssl: False

# Defaults.
use_token: False
verbose: False
certificate: 
format: human
color: True
description_on: False

Could it be a permissions error? When I attempt to run the script at the CLI as a non-root user I get this error: "/opt/rh/python27/root/usr/bin/python2: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory"

Re: Launching tower-cli playbook from Splunk alert

renjith_nair — Tue, 26 Jun 2018 02:12:36 GMT

Yes it could be a permission issue as well as mentioned earlier. sudo as splunk user and try to execute the script