Key Takeaways
- Automated Root Cause Analysis: The AI Troubleshooting Agent proactively analyzes metrics, logs, and traces to pinpoint the root cause of service or infrastructure failures before a human responder even opens the alert.
- Guided Remediation: The Remediation Plan bridges the gap between discovery and resolution by providing step-by-step, in-context instructions.
- Reduced MTTR (Mean Time to Resolution): By automating the "heavy lifting" of investigation, teams can resolve incidents significantly faster, reducing downtime and operational pressure.
- Lower Cognitive Load: These tools minimize manual toil and on-call burnout, allowing developers and SREs to focus on innovation rather than repetitive troubleshooting tasks.
- Versatile Application: The solution is effective across diverse environments, from standard application service errors to complex Kubernetes infrastructure issues like
ImagePullBackOff.
What are the AI Troubleshooting Agent and Remediation Plan?
The AI Troubleshooting Agent and Remediation Plan are built into Splunk Observability Cloud and automate the processes of troubleshooting and resolving both application and Kubernetes related issues. The Troubleshooting Agent uses AI to automatically pinpoint suspected root causes and summarize incidents, and then the Remediation Plan uses the output from the Troubleshooting Agent to provide guided, in-context steps to resolve issues faster and more accurately.
How are they helpful?
If you’re currently in software development, site reliability, or DevOps, you’ve most likely felt the impact of AI – increased efficiency, productivity, and resiliency with decreased manual effort. The AI Troubleshooting and Remediation Plan both help in the same ways – reducing toil, automating and streamlining troubleshooting to reduce downtime, effort, on-call pressure, and mean time to resolution (MTTR).
The AI Troubleshooting Agent acts as an automated first responder that performs the heavy lifting of issue investigation. It analyzes data that exists within Splunk Observability Cloud (metrics, logs, traces) to pinpoint exactly why a service or infrastructure is failing. Is your checkout service throwing 500s in the payment gateway because of a database connection timeout? When the on-call developer gets a page for this, the AI Troubleshooting Agent has already automatically kicked off the troubleshooting process. By the time a real human opens the alert in the UI, they’re fully equipped with suspected root cause(s) of the issue.
The Remediation Plan will then walk through the steps of resolving the issue to bridge the gap between discovery and remediation. Would the issue be resolved by rolling back a Kubernetes deployment, toggling a feature flag, or increasing pod memory limits? The Remediation Plan will suggest specific, step-by-step actions like these.
Using the AI Troubleshooting and Remediation Plan reduces cognitive load, toil, time to issue resolution, and provides all members of development teams with improved context, insight, and actionable resolution steps fast.
Let’s jump in to see how this all works.
How it works
From the home screen of Splunk Observability Cloud, we can see an alert related to our application’s Payment Service:
If we open that alert, we can see more details about the spikes in error rates:
When we scroll down, we can see that the Troubleshooting Agent has automatically started to analyze the issue by looking at metrics, related logs, and other data that might help pinpoint the issue:
Within moments, the AI Troubleshooting Agent completes its investigation, and provides the suspected root cause(s):
It looks like this service is experiencing a high error rate due to HTTP 500 Internal Server Errors originating from the placeOrderHandler function. We can also see that the errors indicate authentication issues in our Payment Service.
So without having to do any manual work, and within a few minutes, we’ve completed troubleshooting and know exactly where a likely root cause of the issue is.
Now because we’re curious developers, we might want to dig in more to understand what data was checked to arrive at the suspected root cause and explore the specific data that was used to determine the root cause. To do this, we can expand the items in the What we’ve checked section:
We can take it further by interrogating the system and asking our own questions by selecting Chat to learn more at the top of the page. This allows us to ask the Splunk Observability Cloud’s AI Assistant in-context questions:
Thanks to the AI Troubleshooting Agent, we know the root cause of the issue we were paged for. Now what?
We can easily get guidance on how to fix the issue by selecting Start action Plan next to our error:
This kicks off the Remediation Plan, which uses the output from the Troubleshooting Agent and creates an action plan for us to use to fix the issue:
Now this one is simple, because we can quickly identify from the provided API key that this is a test token and is not valid in our Production environment. We can deploy a fix with the correct API key, and the issue is resolved. But if we had a more complex issue?
Remediation Plan in a Kubernetes environment
Say that instead of receiving a Payment Service High Error Rate alert, we were alerted on a failed Kubernetes Pod Status. We would still have access to all of the AI Troubleshooting analysis that we previously saw, but our action plan would look a little different:
Here, we see a workload graph on the left and suggested steps to take on the right complete with various kubectl commands to run for more info about the issue.
We can run these commands in our environment, and then paste the results in the Paste the output to determine next steps window below:
The Remediation Plan can then review the output and suggest additional steps to take, if necessary:
And we can again copy the results of running the suggested commands from our terminal and paste the output below:
And after we hit Next, we’re asked to verify if the image exists:
In this case, we’ve discovered that the image doesn’t exist, so we’ll select No – image doesn’t exist or unsure.
We’re then provided with instructions to remediate the image not found issue. We can run this command and set our specific image name:
And then run the additional commands we’re provided in order to verify whether the remediation steps were successful:
Again, after executing the commands we can paste the output below:
And success 🎉 – with a full summary of remediation actions taken and the outcome of fixing the ImagePullBackOff issue! We also get some helpful suggestions so we can avoid similar issues in the future:
Wrap up
Bottom line? The Splunk AI Troubleshooting Agent and Remediation Plan are your new teammates when it comes to quickly and easily troubleshooting and resolving issues. Automating the troubleshooting heavy lifting and providing a clear, guided path to resolution turns hours of manual digging into minutes of actionable insights… while reducing the cognitive load on your team, eliminating on-call burnout, and getting back to building and innovating.
Ready to increase your productivity and reduce downtime? Splunk AI Troubleshooting Agent and Remediation Plan are currently available in US1 and are not available by default. Contact our sales team to try it out today.
New to Splunk Observability Cloud? Start with a free 14-day trial.
Resources
