topic Re: Problem with alert-triggered scripts for ServiceNow in All Apps and Add-ons

Problem with alert-triggered scripts for ServiceNow

gordo32 — Tue, 29 Sep 2020 17:11:54 GMT

I am trying to get the alert-triggered script working but having some difficulties as I keep getting exit code 1 on the scripts.
I'm not a python guy, so I'm unable to reverse-engineer the script, so hoping someone here can assist.

I've installed the Splunk TA for ServiceNow, and configured the logon creds (setting logging to DEBUG)
I have not configured any tables to be pulled down because I'm looking to push only.
I ran the sample query from the documentation and it creates an Incident ticket in ServiceNow successfully:

| snowincident --category "Software" --contact_type "Phone" --subcategory "Database" --short_description "CPU usage is high" --ci_identifier "8214eb87c0a8018b7bd0919758dcc3c2" --priority 1 --splunk_url "hxxp://localhost:8000"

I copied the $SPLUNK_HOME/etc/apps/Splunk_TA_snow/bin/snow_incident.py script to $SPLUNK_HOME/etc/system/local/bin/scripts
Have this alert setup to run every N minutes:

This returns the following when run manually (added commas to improve legibility): *category, contact_type, short_description * network, endpoint_security, Excessive timeouts (138) on www[dot]site[dot]com in the last hour

Now, the alert fires, and calls the python script, but:
a) There is never any debug output. I did a search for "eventtype=snow_*" over "All Time" and there are no results, so I must be failing long before the script gets to any significant portion
b) looking through the _internal logs (e.g. index=_internal snow) I see "runshellscript" instances execute passing the results.csv.gz
c) I get this error message:

ERROR script ... command="runshellscript", Script: /opt/splunk/bin/scripts/snow_incident.py exited with status code: 1

Other things I've tried:
- copying $SPLUNK_HOME/etc/apps/Splunk_TA_snow/bin/*.py to /opt/splunk/bin/scripts
- copying $SPLUNK_HOME/etc/apps/Splunk_TA_snow/bin/script/snow_incident.py to /opt/splunk/bin/scripts
* (as an aside, it's kinda dumb to have 2 separate scripts with different content named snow_incident.py in this TA) *

No matter what I do, I get the status code:1 result.

BTW, in case it matters, I'm running Ubuntu 16.04 and Splunk Enterprise 6.6.4

Any help is appreciated...

Re: Problem with alert-triggered scripts for ServiceNow

gstefancyk — Tue, 29 Sep 2020 17:12:36 GMT

Hi Gordo32,

Have you tried creating your alerts under the context of the Snow app and triggering the script from the default location? I had a similar issue trying to move the script to another location so I ended up just building my searches/alerts under the Splunk_TA_Snow app.

Hope that helps.

Re: Problem with alert-triggered scripts for ServiceNow

gordo32 — Sun, 17 Dec 2017 14:23:32 GMT

Finally found some time to test - and that solved it. Thanks.

Re: Problem with alert-triggered scripts for ServiceNow

PriyankaArivala — Fri, 02 Feb 2018 05:58:02 GMT

Facing same issue. Can you please tell how did you solve that issue?

Re: Problem with alert-triggered scripts for ServiceNow

gordo32 — Fri, 02 Feb 2018 21:10:22 GMT

Yes, as gstefancyk pointed out, after moving the Alert from being under the security context of the Search app over to the context of the SNOW add-on resolved the issue.

Re: Problem with alert-triggered scripts for ServiceNow

gcato — Tue, 29 Sep 2020 18:54:09 GMT

Recently had the same issue and this solution worked - configuring an alert under the Splunk_TA_snow app to send an alert to SNOW (Splunk v6.4.8).

N.B. For notification throttling to work I needed to use the snow_incident.py script instead of the snowincidentstream search command - which will, understandably, always alert in a saved search when search criteria matched.

Anyway, I wanted my alerts configured under their own app so tried softlinking the Splunk_TA_snow/bin directory into my app's directory and, "voila", my app's scripted SNOW alerts started working.

ln -s ~splunk/etc/apps/Splunk_TA_snow/bin ~splunk/etc/apps/<myappname>/bin

If you could be bothered you could probably isolate the necessary Splunk_TA_snow/bin files to a smaller selection and just copy (or softlink) the ones you need into your app's bin (and bin/scripts) directory, but in my case I did not have a bin directory so softlinking the whole Splunk_TA_snow bin works well for me. Also means any Splunk_TA_snow app upgrades should just work.

Hope someone finds this useful too.

Re: Problem with alert-triggered scripts for ServiceNow

santosh_sshanbh — Wed, 06 Feb 2019 12:28:18 GMT

Is there any way to create SNOW incidents without the use of Service NOW add-on? I want to use the REST API's exposed by SNOW to create the incident but not sure of to call them via alert action. Any comments on this topic would be of great help.