All Apps and Add-ons

Splunk Add-on for ServiceNow: How do I set up a custom alert for creating incidents in ServiceNow?

vamsi92
Explorer

Hi there,

I have installed Splunk add-on for ServiceNow in my Splunk Enterprise (Linux)
I am able to create incident in servicenow with |snowincident command in search manually along with Splunk drilldown.

Now I want an incident to be created if one more sources is added into data.
I currently have 9 sources in Splunk (as per data summary) and the search source=* | dedup source also returned 9 results. I have saved the search as an alert and provided trigger condition search results >9, so if I add a new data source, the search result will be 10 so an alert should be triggered. I have indicated it to run a scheduled search every 1 hour. I have selected in action " service now incident integration" and filled the fields like state, category description, correlation id etc..

However, no incident is created in Service Now after adding a data source after configuring the alert and if I follow this alert method to auto generate incidents, where can I see option to configure Splunk drill down visible in Service Now?
Also, is there a way to pass this count of search results to service now ( like 9 results found)

pls help me out.

0 Karma

raja8220
New Member

If i install the splunk plugin in servicenow then can i see the custom fields in splunk alert option ???

0 Karma

jkat54
SplunkTrust
SplunkTrust

Based on your comments, it is my belief that you're creating the alert in the wrong splunk application. Please create the alert under the splunk addon for service now app instead of the searching and reporting app or any other splunk app.

The search that drives your alert is using knowledge objects that are only available in your splunk addon for service now app, and thus when you execute the alert/search under another app, you're not seeing any results.

0 Karma

vamsi92
Explorer

Hi jkat,
I double checked the alert, its created using the splunk aad-on. when i run the search results are popping. then when i save it as alert, tried with both scheduled to run every 1 hour and real time monitoring on alert and give alert condition and alert action as servicenow incident manager and an email.
so when condition meet,alert should be triggered and incident should be created in service-now and also an email should be sent. But i am neting getting a email or an incident. and when i check the alert there the status is no alerts fired.
as per the search command "source=* | dedup source " with scheduled 1 hour. it should show if any new source is added in last 1 hour.
when i am searching after adding a new source its showing 1 source added in last 1 hour but alert is not triggered. I have tried alert condition "no of soruces >0" and "no of results >0" as i am testing with range 1 hour and 1 source added.
I have explained in detailed so u can understand problem better.
thank you

0 Karma

jkat54
SplunkTrust
SplunkTrust

When you save it as an alert, what application is it saving the alert under?

Try running this command and then showing us the saved search (alert):

| rest /servicesNS/-/-/saved/searches splunk_server=local | table id eai:acl.app

0 Karma

vamsi92
Explorer

Hi Jkat, Our linux server is down so sry for delay.
I tried running the command and checked the alert is created under splunk_TA_snow (splunk addon for service now).
When i ran the same alert in search and reporting app in my local host windows setup i can see an an action triggered due to to new source adding.
as i said my main splunk is in linux version running on linux version, I repeated the same thing in linux platform but no event triggered.
and even in local windows setup alert is triggered but alert action I conf is email, and email is not sent to recepient configd. Is there any thing we need to do with settings->server settings-> email settings. ?
Pls help.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Can you please provide a copy of your savedsearches.conf from the splunk_TA_snow addon?

0 Karma

vamsi92
Explorer

In the default folder only 4-5 searches are available. and the local folder's savedsearches.conf is empty. I had saved it as an alert. Is there any specific location i need to search this file for?
but i can see the alert i created under splunk_ta_snow in the availabe saved searches by using the command you have posted earlier "| rest /servicesNS/-/-/saved/searches splunk_server=local | table id eai:acl.app "

0 Karma

jkat54
SplunkTrust
SplunkTrust

I just want to see the syntax used in the file and make sure there isnt a mistake in capitalization, etc.

0 Karma

jkat54
SplunkTrust
SplunkTrust

If the "Alert is created under splunk_TA_snow", then it should be in savedsearches.conf in the same app.

0 Karma

vamsi92
Explorer

the savedsearches.conf file in the path splunk_TA_snow/local is empty and
the savedsearches.conf file in the path splunk_TA_snow/default is having the content :
[ServiceNow CMDB CI Services]
disabled = 1
action.email.reportServerEnabled = 0
action.email.useNSSubject = 1
alert.track = 0
cron_schedule = 0 * * * *
description = Saved search which populates the CMDB CI Relations from ServiceNow
dispatch.earliest_time = 0
dispatch.latest_time = now
display.general.type = statistics
display.visualizations.show = 0
enableSched = 1
request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = eventtype=snow_cmdb_ci_service | dedup sys_id | fields - _bkt, _cd,_indextime,_kv,_raw,_serial,_si,_sourcetype,_subsecond, punct, index, source, sourcetype | inputlookup append=t cmdb_ci_service_lookup | dedup sys_id | outputlookup cmdb_ci_service_lookup

[ServiceNow Sys Choice List]
disabled = 1
action.email.reportServerEnabled = 0
action.email.useNSSubject = 1
alert.track = 0
cron_schedule = 0 * * * *
description = Saved search which populates the sys choice list from ServiceNow
dispatch.earliest_time = 0
dispatch.latest_time = now
display.general.type = statistics
display.visualizations.show = 0
enableSched = 1
request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = eventtype=snow_sys_choice_list | table name, element, value, sys_id | inputlookup append=t sys_choice_list_lookup | dedup sys_id | sort + name, element | outputlookup sys_choice_list_lookup

[ServiceNow Incident State]
disabled = 1
action.email.reportServerEnabled = 0
action.email.useNSSubject = 1
alert.track = 0
cron_schedule = 0 * * * *
description = Saved search which populates incident state from ServiceNow
dispatch.earliest_time = 0
dispatch.latest_time = now
display.general.type = statistics
display.visualizations.show = 0
enableSched = 1
request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = eventtype=snow_sys_choice_list name="incident" element="state" | eval incident_state_name=label | eval state=value | eval incident_state=value| dedup state, incident_state, incident_state_name | table state, incident_state, incident_state_name| inputlookup append=t incident_state_lookup | dedup state | sort + state | outputlookup incident_state_lookup

0 Karma

jkat54
SplunkTrust
SplunkTrust

Hey, you're not marking the answers to your questions as answers. Instead you're up-voting them. Please mark them as answers if they answer your question and upvote answers to other people's questions that helped you.

For example. If someone solves this problem for you, you mark it as the answer. Then if i find this when I have the same issue in the future... and I like the answer that you marked as the answer... then I will upvote the answer.

As for the answer to this question(s). You can view the "status" of the alert in the job inspector. Please use the job inspector to see if the alert was triggered. http://docs.splunk.com/Documentation/Splunk/6.2.0/Knowledge/ViewsearchjobpropertieswiththeJobInspect...

vamsi92
Explorer

Hi jkat,
As per my question with that configuration, will adding a source trigger the incident?
i viewed in job inspector its showing its running but no events fired.
and in the splunk also its showing no events fired. How to trigger the alert to make it create an incident.
and where can we configure splunk drill down ?
before i was using for manual incident creation
| snowincident --category "Software" --contact_type "Phone"
--subcategory "Database" --short_description "CPU usage is high"
--ci_identifier "8214eb87c0a8018b7bd0919758dcc3c2" --priority 1 --splunk_url "http://localhost:8000"
--comments "This is urgent and blocking, can somebody take a look ?"
--correlation_id "de305d51-15b4-411b-adb2-fb6b9e546013"

i was able to create incident and i was able to configure splunk drilldown url too.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...