Help with Retrieving Bulk Incident Data via REST API and Customizing Webhook Notifications in Splunk

[jagan_jijo]

Hi everyone,

I'm working on improving our incident response and monitoring setup using Splunk, and I have a few questions I hope someone can help with:

1. Bulk Incident Data Retrieval During Downtime:
   What’s the best way to retrieve a large volume of incident (via REST API) data from Splunk for a specific timeframe, especially during known downtime periods? Are there recommended search queries or techniques to ensure we capture everything that occurred during those windows?

2. Querying Individual Event Data via Endpoints:
   How can we query Splunk endpoints (e.g., via REST API) to retrieve detailed data for individual events or incidents? Any examples or best practices would be greatly appreciated.

3. Customizing Webhook Notifications:
   Is it possible to modify the structure or content of webhook notifications sent from Splunk without using third-party apps like Better Webhooks or Alert Managers? If so, how can this be done natively within Splunk? 

Thanks in advance for any guidance or examples you can share!  Splunk Enterprise 6.2 Overview REST Endpoint Examples 

[jagan_jijo]

@livehybrid Thanks for the response! We're fairly new to Splunk and currently exploring Enterprise Security (ES) as our primary platform. what do you recommend as the industry standard? ES or ITSI?

We’ve already reviewed the REST API documentation for retrieving fired alerts, search jobs, and events, and just wanted to double-check if that’s the recommended approach for pulling incident data during specific timeframes. Our main goal is to retrieve all incidents that occurred within a defined window and then collect the associated raw events for those incidents.

We’re also exploring the use of HTTP notifications to reduce the number of API queries—ideally by triggering event collection based on incoming alerts.

Regarding Better Webhooks, we’ve looked into it and it seems like a great solution. However, we’re hoping to build something similar natively within Splunk. Do you have any recommendations on how to approach building a custom webhook app or alert action? Also, is there a way to test such an app effectively within Splunk?

[livehybrid · Accepted solution]

Hi @jagan_jijo 

Both ES and ITSI have their own use-cases and strengths. They can also exist together in the same Splunk deployment but ultimately ITSI is used for IT Operations Monitoring (e.g. alerting based on availability of services, Key Performance Indicators etc - whereas ES is all about Security Monitoring.

If you're looking at pulling ES incidents then there is an additional set of APIs that you can make use of (see https://docs.splunk.com/Documentation/ES/8.0.40/API/AboutSplunkESAPI) 

What is the system you are looking to integrate with here? 

The Better Webhooks is just a free app which can be installed within your Splunk environment, just like a custom webhook app would, however there isnt anything stopping you from building your own Splunk alert action custom app to do the same thing if you dont want to use the community-built app. https://dev.splunk.com/enterprise/docs/devtools/customalertactions/ is a good starting point for building a custom alert action - which has a Slack alert example that you might be able to modify. Alternatively you could download the Better Webhook app to see how that is coded and build as required. 

Just for clarity, the Better Webhook app would be as "native" within Splunk as a custom webhook app would be, both would tie in to the alert action framework, it isnt something you have to host separately.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[livehybrid]

Hi @jagan_jijo 

Please could you provide a little more information on your usecases here and what kind of data you are looking to extract from Splunk? 

You can download data using the search REST API - Check out the following page on how to execute searches using the REST API: https://docs.splunk.com/Documentation/Splunk/9.4.2/RESTTUT/RESTsearches

Regarding pulling data on specific incidents, are you using IT Service Intelligence (ITSI) or Enterprise Security (ES) which has your incidents collated? There are specific endpoints for these premium apps to provide things like incidents/notable events etc depending on your use-case. 

Regarding webhooks, the native webhook sending is quite limited (see https://docs.splunk.com/Documentation/Splunk/9.4.0/Alert/Webhooks) - I'd usually recommend looking at Better Webhooks on SplunkBase, is there a particular problem you're having with that app?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing