Splunk IT Service Intelligence: How to extract the underlying KPI details and pass to the notable event

[raynold_peterso]

We have configured ITSI with entities and services for our application. We have multiple services which all work together which are the guts of our application.

I have created Correlation searches to gather together the like alerts into groups, such as db garbage collection, MQ queue depth, etc. Along with the Correlation searches, I created the Notable Event Aggregation policies we are using for reporting and the like.

Once an event/alert is detected, we push this info to OpsGenie. All of this is working as expected, except for one thing. The Alerts hitting OpsGenie don't contain any information about the KPI's and Entities which originally triggered the event. All I am getting in OpsGenie is the description of the grouped events out of the Notable Event. This is a very generic message and not very helpful.

Now, if you look at the Notable Event, you will see the KPI's assigned to the triggered group along with the services impacted. That is the data I would like to push through to OpsGenie.

I look under the Grouped Events tab in the Notable event and then drill down to one of the alerts details. I would think this is where I could use some form of field substitution to alter the description. But.... The details I want are not there. Well, they are there, but its in the form of field id's and the like.

I am sure there is a way to alter the correlation search to enrich my data to pass it along to OpsGenie. OpsGenie has several unused fields that it can pull from Splunk to supply my level of detail needed. The only problem is those fields do not exist yet in the Notable Event.

So, to my question. If I want to add the alerting KPI's, along with the correlating Entities, to my Notable Event data, how would I go about and accomplish that task.

Don't beat me up to bad, I'm just a fellow trying to learn.

Thanks in advance,
Rcp