Alerting

Creating ServiceNow Incidents via Splunk Enterprise Alerting?

astackpole
Path Finder

Hello Fellow Splunkers!

The goal is to create ServiceNow Incidents/Events exclusively from Splunk Enterprise alerts using the Custom Alert action (we do not have Splunk ES or Splunk ITSI*). 

I have a distributed Splunk Enterprise deployment that contains an Indexer Cluster, Heavy Forwarder, and two Standalone Search heads (in addition to the Cluster Master and Deployment Server).

I have yet to see this implementation work in a deployment with only Splunk Enterprise. Please let me know if this configuration is possible with an on-prem Splunk Enterprise deployment. 

For context, I currently have the following configured,

  1. Splunk_TA_snow deployed to the Search Heads, Heavy Forwarder and Indexer Cluster
    (the add-on in the Indexer Cluster does not contain the inputs.conf file)
  2. Logs are being ingested via the Heavy Forwarder and the ServiceNow account is making successful connections to the Heavy Forwarder and Search Heads configured account.
  3. I have tried configuring the below on alerts with no luckScreen Shot 2022-05-03 at 9.16.19 AM.png
  4. I have also tried passing | snowincident within the alert's SPL to create a new incident in SNOW.

Any help or tips will be greatly appreciated!

Labels (2)
0 Karma

Roy_9
Motivator

Hello @astackpole 

As you said, you installed Servicenow TA you will see 2 options under alert actions you can try create servicenow Incident integration where you need to provide the account(we have used a service account for this provided by servicenow team) and endpoint as /api/now/table/incident and rest of all the fields as per your choice.this is currently in place in our environment up and running.

 

Btw we have the alerts created on SH and this add-on got installed on SH.

Hope this info helps and accept this as a solution if it worked for you.

 

Thanks

0 Karma

PaulPanther
Builder

Hello @Roy_9 how do you handle the incident correlation? Do you have a smart solution to either prevent mutiple incidents for the same issue and create new incident if the previous created incident based on the same alert  has been already set to resolved?

0 Karma

LearninStuff
Explorer

Apologies -- I overlooked something in your reply.  There are two alert actions provided by the TA:  Event Integration and Incident Integration.  The Incident Integration action is what you need if you are wanting to create incidents directly from Splunk.  The Event Integration, as you have found, creates Splunk events in ServiceNow.  There will need to be a ServiceNow workflow set up in order for those events to be upgraded to incidents.  We opted for using the Incident Integration.

 

0 Karma

LearninStuff
Explorer

Also, you mention ingesting logs from ServiceNow via your HF but you don't mention using any of the inputs to ingest data from ServiceNow tables.  Is the account you have configured able to hit the ServiceNow REST API?

0 Karma

astackpole
Path Finder

Yes, we are currently ingesting SNOW information into Splunk and are able to hit the REST API. It's the Splunk alerts themselves we're unable to get into ServiceNow in the appropriate formatting we want (those are being sent from the Search Heads since we have a distributed deployment). They currently show up in the 'Splunk Import Set' as shown below. However, we need these to be true Splunk Incidents/Events - I believe there's a way to use this Import Set to do so? but that's the piece we're currently missing.

Screen Shot 2022-06-15 at 5.41.13 PM.png

Once they show up as true alerts and not just part of an Import Set, I assume they will begin functioning as normal Events/Incidents to create actionable tickets versus just being extra information stored in ServiceNow (or at least that's the end goal). I have much less experience in ServiceNow compared to Splunk, so any help around transforming this Import Set into Events/Incidents will be greatly appreciated! 

0 Karma

LearninStuff
Explorer

What roles does your splunk account have in ServiceNow?

0 Karma

LearninStuff
Explorer

Hi!  We've been using the ServiceNow TA in a variety of scenarios.  What are you getting when you try the scenarios you describe?

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...