All Apps and Add-ons

ServiceNow Add-on Managing Events

brdr
Contributor

Hello,

We are integrating Splunk and ServiceNow. We use alert triggers that call the ServiceNow Event Integration trigger (calls snow_event.py. Note we don't presently use snow_incident.py). It is easy enough intergration to send and alert (an event) to ServiceNow system. However, when the alert clears on Splunk we want to clear (close) the event on ServiceNow. How can we in Splunk manage the closing of the previous fired alert when the alert goes away? For example:

We want to send an alert (event) to serviceNow from Splunk when a node (host) has high CPU utilization (severity = critical). When the CPU usage goes back to normal we need to figure out how to send another event into ServiceNow with serverity=0 to resolve the associated incident and closes the alert in Service Now. Does the ServiceNow add-on provide a way to do this? I've gone through documentation and couldn't find a way to do this. Has any one figured out a way to manage the state (using severity) of the alert from Splunk to Service now after event creation?

Thanks for your help!
brdr

stboch_umd
Explorer

BRDR,

It sounds like what you are looking for is the snoweventstream.

http://docs.splunk.com/Documentation/AddOns/released/ServiceNow/Usestreamingcommands

The following example search creates an incident when CPU usage is 95 or higher.

sourcetype="CPURates" earliest=-5m latest=now 
| stats avg(CPU) as CPU last(_time) as time by host 
| where CPU>=95 | eval contact_type="email" 
| eval ci_identifier=host | eval priority="1" 
| eval category="Software" | eval subcategory="database" 
| eval short_description="CPU on ". host ." is at ". CPU 
| snowincidentstream
The following example search closes the above incident in ServiceNow version Eureka when CPU usage drops below 15.

sourcetype="CPURates" earliest=-5m latest=now 
| stats avg(CPU) as CPU last(_time) as time by host 
| where CPU<15 | eval contact_type="email" 
| eval ci_identifier=host | eval state="7" 
| eval category="Software" | eval subcategory="database" 
| eval short_description="CPU on ". host ." is at ". CPU 
| snowincidentstream
0 Karma

jkat54
SplunkTrust
SplunkTrust

When you create an event with the Alert action it should log the response somewhere in splunk.

The response should have the service now incident / sr/ Cr that you’ve created.

So you would need to run two searches, one that creates the alerts and one that clears the alerts if certain conditions are met and also appends the results of the previous alert so that it has the inc/sr/cr

Something like

...
| eval severity=if(_time>=now()-900 AND (isnotnull(CR) OR isnotnull(SR)),0,1)
...

darrenfuller
Contributor

So , this is not a question of how to update an existing event, this is a question of how to update specifically the severity of the existing event in SNow when the threshold has been un-crossed ... yes?

0 Karma

brdr
Contributor

yes, that is correct. I need a way in Splunk to generate a severity 0 event against a previously fired that has cleared in Splunk.

0 Karma

darrenfuller
Contributor

OK, and are you creating your alert with the snow_event.py alert action , or with the snowevent command?

0 Karma

brdr
Contributor

we are using the snow_event.py alert action.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...