Hourly Test

[Matt_Splunk]

Hello all,

I am very new to Splunk so apologies if this is an easy one. 

We want to set up an alert to come out of Splunk into ServiceNow to show that the integration is working, however this would inflate our real incident numbers as its an extra 24 false incidents per day.

What can we set up to know that the integration is broken without having to keep checking it manually? The current environment isnt massively noisy so it could be while before anyone notices that something is wrong.

Any feedback much appreciated, Thank you.

[sumansah]

You can catch a broken integration without creating test incidents: alert on the add-on's own errors. If you still want a positive "it works" signal, send the heartbeat to ServiceNow as an event rather than an incident.

1. Alert on failures (no test traffic needed)

The Splunk Add-on for ServiceNow writes its logs to _internal. The add-on troubleshooting docs give this search for them:

index=_internal source=*snow*

Once the add-on is installed, a scheduled alert on errors in those logs is a good starting point:

index=_internal source=*snow* ERROR
| stats count latest(_raw) as last_error by source host

Trigger when the number of results is greater than 0. Treat the ERROR filter as a first pass. Once you're up, look at what the add-on actually logs when a send fails (bad credentials, timeouts, HTTP errors), e.g. by sending a test alert to a deliberately wrong account on a non-production stack, and tighten the filter to match.

Splunk also has Settings > Alert actions > View log events for each alert action, which is a quick way to see what the ServiceNow action is logging.

2. Optional heartbeat that doesn't inflate incident counts

The add-on's alert actions can create both incidents and events in ServiceNow. Sending the hourly heartbeat as an event proves the path works end to end without touching incident numbers. Check with your ServiceNow team that no event rule turns these heartbeats into incidents. "No heartbeat event in the last ~2 hours" then becomes the "integration is down" check on the ServiceNow side.

3. Route the failure alert outside ServiceNow

As yuanliu pointed out, if the ServiceNow path is broken, an alert sent through ServiceNow won't arrive. Use email or another channel for this one.

Since you're still designing it: give the ServiceNow integration account only the permissions it needs, and keep its credentials in the add-on's configuration, not in searches or scripts.

This works the same on Splunk Enterprise and Splunk Cloud; the add-on supports both.

Docs:

• Splunk Add-on for ServiceNow (Splunkbase)
• Troubleshooting Splunk supported add-ons (log searches per add-on)
• Using the alert actions manager

Drafted with AI assistance and reviewed by me — please verify in your environment before applying.

S Sah| Keep Splunkinggg...

[Vikas-Jha]

It would be better to create some alert on the error logs to get notified of broken integration. 

[livehybrid]

Hi @Matt_Splunk 

Which TA are you using? I would investigate the ability to update an existing incident rather than create a new one, or some other interaction using the app which proves the connectivity is in place and work, ie a custom command which searches or loads a SNOW event/incident?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful

• Marking it as the solution if it resolved your issue

• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

[Matt_Splunk]

Hi @livehybrid ,

We haven't deployed Splunk yet, so we haven't installed the ServiceNow integration at this stage. We're currently in the planning/design phase and are looking for recommended approaches to monitoring the Splunk to ServiceNow integration without generating additional incidents that would skew incident metrics. Any general best-practice guidance would be appreciated or is there an alert that could be set up if "x" conditions are met that would show the link is broken

[yuanliu]

Let me see if I get the requirement right: You want to construct a synthetic loop to detect whether Splunk alert -> ServiceNow incident pipeline is broken.  If this is correct, my first reaction is: What are you going to do if someone sees that Splunk can no longer generate incident in ServiceNow? (Assuming that ServiceNow is your incident management platform.)  If Splunk can no longer generate incident, there is a good chance that no one can. (I guess you can make this alert use a different notification channel, and not use ServiceNow.)

Logistical challenge aside, I have an operational question.  Why 24 false incidents is bad?  Everyone would understand that these are synthetic test incidents.  Any meaningful reporting in ServiceNow should automatically exclude them. (If your reporting is in Splunk, you can just as easily filter them out.)  As to incident handling, nothing prevents you from setting up an automatic workflow in ServiceNow to close such synthetic incidents as "Not an issue" and no worker needs to see them.

[gcusello]

Hi @Matt_Splunk ,

Use the Splunk Add-On for SERVICENOW (https://splunkbase.splunk.com/app/1928).

It will give your all the features you need.

Ciao.

Giuseppe