How to debug Splunk alert scripts?

[markb_1984]

I have a script which takes all the 8 splunk args, extracts the csv containing the event details and then does some transformation before sending an alert to an IM.

To improve its reusability, I split out various parts of the script into separate shell scripts all located within a subdirectory of the main script splunk executes. Having completed is refactoring, I have successfully executed the main script passing in dummy (but sensible) values for all the expected args. Splunk is however is not successfully executing this main script.

I have followed most of the suggestions in the following wiki to no avail: http://wiki.splunk.com/Community:TroubleshootingAlertScripts

Increasing the logging to error in runshellscript.py does then reveal an error entry (in python.log) for my script when the alert fires. The error however simply outputs all the args that are passed down to the shell script. No other error is reported in this log.

Is there anywhere else I can look to see errors that might be emanating from splunk executing my script?

[sjscott]

The easiest way to debug your scripts is to put a launcher in front of it. The idea is to redirect stdout and stderr to a file when the script runs. Thus allowing you to see what's going on. Below is what I use on RedHat for launching a python script.

#!/bin/bash
#
#  We need to lauch the python script from Bash so we can use the python version within the OS.  This
#  allows us to install the python modules we need from the package manager of the OS.   The python
#  that comes with Splunk most likely won't have the modules you require.  Thus we use the one from
#  the OS.
#
# Debug
# Use the following line to debug.  send stdout stderr to the file specified below
/usr/bin/python /opt/splunk/bin/scripts/<YOURSCRIPT> "$@" >>/var/tmp/file.txt 2>&1

# production
#/usr/bin/python /opt/splunk/bin/scripts/<YOURSCRIPT> "$@"

The "$@" takes the args that were passed to the launcher and re-passes them to the real script. The secondary reason I use a launcher is I don't want to use the version of Python that comes built into Splunk.