All Apps and Add-ons

ServiceNow Add-on Managing Events

brdr
Contributor

Hello,

We are integrating Splunk and ServiceNow. We use alert triggers that call the ServiceNow Event Integration trigger (calls snow_event.py. Note we don't presently use snow_incident.py). It is easy enough intergration to send and alert (an event) to ServiceNow system. However, when the alert clears on Splunk we want to clear (close) the event on ServiceNow. How can we in Splunk manage the closing of the previous fired alert when the alert goes away? For example:

We want to send an alert (event) to serviceNow from Splunk when a node (host) has high CPU utilization (severity = critical). When the CPU usage goes back to normal we need to figure out how to send another event into ServiceNow with serverity=0 to resolve the associated incident and closes the alert in Service Now. Does the ServiceNow add-on provide a way to do this? I've gone through documentation and couldn't find a way to do this. Has any one figured out a way to manage the state (using severity) of the alert from Splunk to Service now after event creation?

Thanks for your help!
brdr

stboch_umd
Explorer

BRDR,

It sounds like what you are looking for is the snoweventstream.

http://docs.splunk.com/Documentation/AddOns/released/ServiceNow/Usestreamingcommands

The following example search creates an incident when CPU usage is 95 or higher.

sourcetype="CPURates" earliest=-5m latest=now 
| stats avg(CPU) as CPU last(_time) as time by host 
| where CPU>=95 | eval contact_type="email" 
| eval ci_identifier=host | eval priority="1" 
| eval category="Software" | eval subcategory="database" 
| eval short_description="CPU on ". host ." is at ". CPU 
| snowincidentstream
The following example search closes the above incident in ServiceNow version Eureka when CPU usage drops below 15.

sourcetype="CPURates" earliest=-5m latest=now 
| stats avg(CPU) as CPU last(_time) as time by host 
| where CPU<15 | eval contact_type="email" 
| eval ci_identifier=host | eval state="7" 
| eval category="Software" | eval subcategory="database" 
| eval short_description="CPU on ". host ." is at ". CPU 
| snowincidentstream
0 Karma

jkat54
SplunkTrust
SplunkTrust

When you create an event with the Alert action it should log the response somewhere in splunk.

The response should have the service now incident / sr/ Cr that you’ve created.

So you would need to run two searches, one that creates the alerts and one that clears the alerts if certain conditions are met and also appends the results of the previous alert so that it has the inc/sr/cr

Something like

...
| eval severity=if(_time>=now()-900 AND (isnotnull(CR) OR isnotnull(SR)),0,1)
...

darrenfuller
Contributor

So , this is not a question of how to update an existing event, this is a question of how to update specifically the severity of the existing event in SNow when the threshold has been un-crossed ... yes?

0 Karma

brdr
Contributor

yes, that is correct. I need a way in Splunk to generate a severity 0 event against a previously fired that has cleared in Splunk.

0 Karma

darrenfuller
Contributor

OK, and are you creating your alert with the snow_event.py alert action , or with the snowevent command?

0 Karma

brdr
Contributor

we are using the snow_event.py alert action.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...