I am trying to get the alert-triggered script working but having some difficulties as I keep getting exit code 1 on the scripts.
I'm not a python guy, so I'm unable to reverse-engineer the script, so hoping someone here can assist.
| snowincident --category "Software" --contact_type "Phone"
--subcategory "Database" --short_description "CPU usage is high"
--ci_identifier "8214eb87c0a8018b7bd0919758dcc3c2" --priority 1
index=waf sourcetype=imperva_incapsula_cef sourceServiceName="www[dot]site[dot]com" | where isnull(cn1) | stats count as Timeouts | where Timeouts>50 | eval category="network" | eval contact_type="endpoint_security" | eval urgency=2 | eval impact=3 | eval short_description="Excessive timeouts (". Timeouts .") on www[dot]site[dot]com in the last hour" | table category, contact_type, short_description
Now, the alert fires, and calls the python script, but:
a) There is never any debug output. I did a search for "eventtype=snow_*" over "All Time" and there are no results, so I must be failing long before the script gets to any significant portion
b) looking through the _internal logs (e.g. index=_internal snow) I see "runshellscript" instances execute passing the results.csv.gz
c) I get this error message:
ERROR script ... command="runshellscript", Script: /opt/splunk/bin/scripts/snow_incident.py exited with status code: 1
Other things I've tried:
- copying $SPLUNK_HOME/etc/apps/Splunk_TA_snow/bin/*.py to /opt/splunk/bin/scripts
- copying $SPLUNK_HOME/etc/apps/Splunk_TA_snow/bin/script/snow_incident.py to /opt/splunk/bin/scripts
* (as an aside, it's kinda dumb to have 2 separate scripts with different content named snow_incident.py in this TA) *
No matter what I do, I get the status code:1 result.
BTW, in case it matters, I'm running Ubuntu 16.04 and Splunk Enterprise 6.6.4
Any help is appreciated...
Recently had the same issue and this solution worked - configuring an alert under the Splunk_TA_snow app to send an alert to SNOW (Splunk v6.4.8).
N.B. For notification throttling to work I needed to use the snow_incident.py script instead of the snowincidentstream search command - which will, understandably, always alert in a saved search when search criteria matched.
Anyway, I wanted my alerts configured under their own app so tried softlinking the Splunk_TA_snow/bin directory into my app's directory and, "voila", my app's scripted SNOW alerts started working.
ln -s ~splunk/etc/apps/Splunk_TA_snow/bin ~splunk/etc/apps/<myappname>/bin
If you could be bothered you could probably isolate the necessary Splunk_TA_snow/bin files to a smaller selection and just copy (or softlink) the ones you need into your app's bin (and bin/scripts) directory, but in my case I did not have a bin directory so softlinking the whole Splunk_TA_snow bin works well for me. Also means any Splunk_TA_snow app upgrades should just work.
Hope someone finds this useful too.
Have you tried creating your alerts under the context of the Snow app and triggering the script from the default location? I had a similar issue trying to move the script to another location so I ended up just building my searches/alerts under the Splunk_TA_Snow app.
Hope that helps.
Is there any way to create SNOW incidents without the use of Service NOW add-on? I want to use the REST API's exposed by SNOW to create the incident but not sure of to call them via alert action. Any comments on this topic would be of great help.