Creating ServiceNow Incidents via Splunk Enterprise Alerting?

[astackpole]

Hello Fellow Splunkers!

The goal is to create ServiceNow Incidents/Events exclusively from Splunk Enterprise alerts using the Custom Alert action (we do not have Splunk ES or Splunk ITSI*). 

I have a distributed Splunk Enterprise deployment that contains an Indexer Cluster, Heavy Forwarder, and two Standalone Search heads (in addition to the Cluster Master and Deployment Server).

I have yet to see this implementation work in a deployment with only Splunk Enterprise. Please let me know if this configuration is possible with an on-prem Splunk Enterprise deployment. 

For context, I currently have the following configured,

1. Splunk_TA_snow deployed to the Search Heads, Heavy Forwarder and Indexer Cluster
   (the add-on in the Indexer Cluster does not contain the inputs.conf file)
2. Logs are being ingested via the Heavy Forwarder and the ServiceNow account is making successful connections to the Heavy Forwarder and Search Heads configured account.
3. I have tried configuring the below on alerts with no luck
4. I have also tried passing | snowincident within the alert's SPL to create a new incident in SNOW.

Any help or tips will be greatly appreciated!

[Roy_9]

Hello @astackpole 

As you said, you installed Servicenow TA you will see 2 options under alert actions you can try create servicenow Incident integration where you need to provide the account(we have used a service account for this provided by servicenow team) and endpoint as /api/now/table/incident and rest of all the fields as per your choice.this is currently in place in our environment up and running.

Btw we have the alerts created on SH and this add-on got installed on SH.

Hope this info helps and accept this as a solution if it worked for you.

Thanks

[PaulPanther]

Hello @Roy_9 how do you handle the incident correlation? Do you have a smart solution to either prevent mutiple incidents for the same issue and create new incident if the previous created incident based on the same alert  has been already set to resolved?

[LearninStuff]

Apologies -- I overlooked something in your reply.  There are two alert actions provided by the TA:  Event Integration and Incident Integration.  The Incident Integration action is what you need if you are wanting to create incidents directly from Splunk.  The Event Integration, as you have found, creates Splunk events in ServiceNow.  There will need to be a ServiceNow workflow set up in order for those events to be upgraded to incidents.  We opted for using the Incident Integration.

[LearninStuff]

Also, you mention ingesting logs from ServiceNow via your HF but you don't mention using any of the inputs to ingest data from ServiceNow tables.  Is the account you have configured able to hit the ServiceNow REST API?

[astackpole]

Yes, we are currently ingesting SNOW information into Splunk and are able to hit the REST API. It's the Splunk alerts themselves we're unable to get into ServiceNow in the appropriate formatting we want (those are being sent from the Search Heads since we have a distributed deployment). They currently show up in the 'Splunk Import Set' as shown below. However, we need these to be true Splunk Incidents/Events - I believe there's a way to use this Import Set to do so? but that's the piece we're currently missing.

Once they show up as true alerts and not just part of an Import Set, I assume they will begin functioning as normal Events/Incidents to create actionable tickets versus just being extra information stored in ServiceNow (or at least that's the end goal). I have much less experience in ServiceNow compared to Splunk, so any help around transforming this Import Set into Events/Incidents will be greatly appreciated! 

[LearninStuff]

What roles does your splunk account have in ServiceNow?

[LearninStuff]

Hi!  We've been using the ServiceNow TA in a variety of scenarios.  What are you getting when you try the scenarios you describe?