How to Send Splunk Observability Alerts to Webex teams in Minutes

[msimon-splunk]

As a Developer Evangelist at Splunk, my team and I are constantly tinkering with technology to explore its nuances and discover practical use cases. Most of that experimentation is short-lived, helping to drive awareness through demos and content. But occasionally, a solution sticks and lives on well beyond the initial project.

A great example is splunkgames.com—a simple single-page app that acts as a portal to interactive experiences we’ve built to help grow awareness of OpenTelemetry, Observability concepts, and Splunk’s Observability portfolio. You’ll even find links to our blogs and YouTube playlists there (if you haven’t checked it out yet, go take a look!).

While we’re not running a battle-tested SRE team for these environments (I know, I know… don’t judge!), we do need to keep the site available and generally responsive. And you guessed it—we leverage SplunkSynthetic Monitoring, part of Splunk Observability Cloud,to keep tabs on the health of the landing page and the availability of the interactive content.

Recently, with Splunk joining the Cisco party, we’ve also gained access to another great collaboration tool: Webex. Since our team primarily communicates through Webex Messaging, I figured—why not integrate Splunk Observability Cloud alerts directly into the channels we already use?

Like many things in tech, I found scattered pieces of the puzzle and stitched them together. A huge shoutout to the ThousandEyes Webex Integration Guide, which—although focused on ThousandEyes—walks through exactly how to configure the Webex side of the integration (and yes, I literally copied-pasta’ed steps 1-5).

Having worked on the Splunk Observability Cloud webhook configurations before—including partnering with the team on the “Splunk Platform” integration—and recalling Caitlin’s excellent blog on sending Observability alerts to Jira, I decided to spend a couple of hours wiring it all together.

The result? A relatively clean integration that drops alerts directly into our Webex Space. My team can see alerts as they fire, chat about them directly in the thread with things like “got it,” “looking into it,” or my personal favorite—“Mike! Not again!”

Quick Template Overview: What This Solution Provides

• Leverages Webex Adaptive Cards for a clean, structured alert view.
• Includes a fallback to a standard text message if Adaptive Cards aren’t supported.
• Uses Splunk Observability Cloud's custom webhook “if” logic to display optional fields like the detector’s Runbook URL and TIP only when they’re defined in the detector configuration.
• Dynamically sets the message’s color using the severityDecoder helper function, aligning severity levels to appropriate visual cues. (See Splunk Webhook Helper Functions for details.)
• Adds an actionable button that drives directly to Splunk Observability Cloud’s alert view—helping expedite troubleshooting right from the message.

When it’s all said and done (correctly) the result looks something like this

Step by Step: Leveraging Cisco Webex for Alert Notifications

This article shows you how to receive Splunk Observability Cloud alert notifications in a Cisco Webex App space using webhooks.

1. Create a new Webex bot: A Webex bot lets you integrate services into Webex. In the Webex App, create a new bot to receive alert notifications from the ThousandEyes webhook. For detailed instructions, see Creating a Webex Bot. Use the screenshots below to help you create your bot.

2. Copy the bot's access token: After setting up your new Webex bot, copy the bot's access token and save it somewhere handy. You'll need this token when you configure the Splunk Observability cloud  integration (step 6)

3. In Webex App, create a space to receive alert notifications from Splunk Observability Cloud: Skip this step if you already have a Webex App space where you'd like to receive alerts.

4. Add the Webex App bot to the space: In this example, the bot named "Splunk Observability Cloud Alerts" is added to the "Splunk O11y Cloud Example Alert Space" Webex App space.. Then Click “Create”

5. Copy the space's roomId: You'll use the roomId with the bot's access token (from Step 2) to configure the ThousandEyes integration. Now that the Webex bot is a member of your Webex App space, look up the roomId of your Webex App space and copy it. To find the roomId:

• Use your Webex App login to access the Webex API site.
• Using a web browser, navigate to the following URL and click “Run” to look up the roomId: https://developer.webex.com/docs/api/v1/rooms/list-rooms

6. Create A new WebHook integration in Splunk Observability cloud. Log into your Observability cloud Organization. Choose “Data Management” from the left nav. Then Click “Add Integration”

Use the Search box and locate the “WebHook” notification services. Click the integration tile to start the setup:

7. Configure the Webhook

           Click Next at the Webhook summary page

Enter the following information into the “Configure connection” form, then click “Next”

Field	Value
Name	Provide a descriptive name. This name will appear in the “Alert recipients” section of your detector configuration
URL	https://webexapis.com/v1/messages
Method	POST
Shared Secret	Leave default
Headers
Key	Value
Content-type	application/json
Authorization	Bearer [token_from_step_2]

Pro-tips:

• Keys and Values may be case sensitive
• In the Authorization header the value should be in the following format “Bearer[space][token_value_from_step2]” (without the quotes)
  • For example; if your token value from step2 was “123456789ABCDEFG” then the value of your Authorization header would be “Bearer 123456789ABCDEFG” (without the quotes)

8. Copy and Paste the template into the “Customize message” payload. Update line 6 with the roomID obtained in Step5 (replace “<YOUR_WEBEX_ROOM_ID>” with your token) then click Next

Custom Payload (copy and paste - remember to remove the angle brackets around your Webex room ID but leave the quotes)

   {{!--
 Webex Room ID: Update this with the target Webex Room ID where notifications should be sent.
 Retrieve the Room ID using the Webex API or Webex Control Hub.
--}}
{
 "roomId": "<YOUR_WEBEX_ROOM_ID>",


 {{!--
   Markdown Fallback: Used if Adaptive Cards aren’t supported by the Webex client.
   Provides a readable plain-text version of the alert details. Best practice is to update this section as well as the adaptive card
 --}}
 "markdown": "<strong><u>{{{encodeString messageTitle}}}</u></strong>\n<strong>Severity:</strong> {{severity}}\n<strong>Status:</strong> {{{statusExtended}}}\n<strong>Source of Alert: </strong>{{{coalesce src 'src not found'}}}\n<strong>Alert Description:</strong>\n{{{messageBody}}}\n<strong>Detector: </strong>[{{{encodeString detector}}}]({{{detectorUrl}}}&orgID={{orgId}})\n{{#if runbookUrl}} [Runbook URL]({{runbookUrl}})\n{{/if}}{{#if tip}} <strong>Detector TIP specified:</strong>\n{{{encodeString tip}}}\n{{/if}}",


 "attachments": [
   {
     "contentType": "application/vnd.microsoft.card.adaptive",
     "content": {
       "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
       "type": "AdaptiveCard",
       "version": "1.3",


       "body": [
         {{!--
           Alert Title with Dynamic Severity Color.
           The severityDecoder helper function is used to determine the color based on severity.
           For more details, see Splunk Webhook Helper Functions:
           https://dev.splunk.com/observability/docs/integrations/webhook_integration_overview#Helper-functions
         --}}
         {
           "type": "TextBlock",
           "text": "{{{encodeString messageTitle}}}",
           "wrap": true,
           "weight": "bolder",
           "size": "large",
           "color": "{{{severityDecoder Critical='attention' Major='warning' Minor='accent' Warning='accent' Info='good' ok='good' default='default'}}}"
         },


         {{!-- Generic Alert Fields: Includes event timestamp, severity, status, source, and description --}}
         {
           "type": "TextBlock",
           "text": "Event Raised at: {{timestamp}}",
           "wrap": true
         },
         {
           "type": "TextBlock",
           "text": "Severity: {{severity}}",
           "wrap": true
         },
         {
           "type": "TextBlock",
           "text": "Status: {{{statusExtended}}}",
           "wrap": true
         },
         {
           "type": "TextBlock",
           "text": "Source of Alert: {{{coalesce src 'src not found'}}}",
           "wrap": true
         },
         {
           "type": "TextBlock",
           "text": "Alert Description:\n{{{messageBody}}}",
           "wrap": true
         },


         {{!-- Optional TIP Section: Included only if 'tip' is present in the detector config--}}
         {{#if tip}}
         {
           "type": "TextBlock",
           "text": "Detector TIP specified:\n{{{encodeString tip}}}",
           "wrap": true
         },
         {{/if}}


         {{!-- Action Buttons: Provide quick access to investigate the alert or view the runbook (if one is defined in the detecor config) --}}
         {
           "type": "ActionSet",
           "actions": [
             {
               "type": "Action.OpenUrl",
               "title": "Investigate in Splunk Observability Cloud",
               "url": "{{{detectorUrl}}}&orgID={{orgId}}"
             }
             {{#if runbookUrl}},
             {
               "type": "Action.OpenUrl",
               "title": "Runbook",
               "url": "{{{runbookUrl}}}"
             }
             {{/if}}
           ]
         }
       ]
     }
   }
 ]
}

Click “Save” at the “Review and save” page

9. Update your Detectors “Alert Receipts” Select “Detectors & SLOs”, then select “Detectors” , identify the detectors you’d like to receive Webex notifications about and select “Manage subscriptions”

Click the “Add recipient" button, choose “Webhook” from the menu, choose your newly created webhook from the drop down, then click the “save” button

10. Once a Detector alerts, you should see an Adaptive card appear in your newly created space

Wrapping Things Up:

Already using Splunk Observability Cloud and Webex Messaging? Why not take it a step further and bring alerts directly into the conversations you’re already having?

If you’re not using either product, why not? Check out Webex and/or start your Splunk Observability Cloud Trial today!

Leverage the template above as-is or customize it to meet your team’s specific needs. It’s a simple way to improve visibility, reduce response times, and (let’s be honest) have a little fun roasting your teammates when things go sideways. Leave a comment below if you use this or found it helpful and stay tuned for more.