This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. Specifically, this series goes into the specific Splunk AI Assistant use cases of:
- Identifying unknown unknowns
- Analyzing and troubleshooting in real-time
- Auditing compliance and cost
- Explaining metrics and providing feedback
- Onboarding new hires or new users of Splunk Observability Cloud
- Observability as Code
If you’d like to start with how to access the Splunk AI Assistant and identify unknown unknowns within a service environment, check out the first post in this series: Identifying Unknown Unknowns. In our second post, Analyzing and Troubleshooting in Real-Time, you can see how to use the AI Assistant to analyze trace errors and database query performance and gain insights into detectors and alerts.
In this third post, we’ll learn how to leverage the Splunk AI Assistant to ensure we remain in compliance with our organization’s specific requirements. We’ll also see how the AI Assistant can analyze our infrastructure and identify areas that may be underutilized, which in turn will help us reduce costs.
Compliance
Organizations often implement code-level tagging requirements for compliance, security, cost, and operations. For example, it’s a requirement that all of the services in our Online Boutique environment set a tag value for the attribute tenant.level to easily distinguish between service tiers. This tenant.level attribute has three tag values – gold, bronze, and silver – as seen here in Tag Spotlight within Splunk Application Performance Monitoring:
We can use the AI Assistant to audit the data in our environment and ensure we are in compliance with this specific organizational requirement. We’ll ask the AI Assistant if any services in the Online Boutique environment aren’t using this required tenant.level tag. In the response, we can see that our Ad service is missing the required tenant.level tag:
Note: In the AI Assistant pane, if you have “Use current page filters” toggled on and you try to run this query from a specific page, like the paymentservice detailed view, you might not see the full results you expect. Querying from this page would set the context to just that of the paymentservice. If we want to interrogate our entire environment and all services instead, we would need to move up a level and execute the query from our services view that lists and sets the context as all services in our environment.
If we go to our Ad service in Splunk APM, we can verify that the data from it is in fact missing the required tenant.level tag:
Cost Control
Similar to what we saw in our last post about reducing alert noise and ecosystem hygiene, we can use the AI Assistant to analyze our infrastructure and help us identify areas that might not be efficiently utilized. This can help us capacity plan and reduce costs when allocating infrastructure.
We’ll have the AI Assistant analyze our Kubernetes infrastructure by prompting it to tell us whether it’s underutilized and identify nodes in Splunk Infrastructure Monitoring with all-time high CPU usage below 90% in our Online Boutique environment.
The AI Assistant’s analysis identifies a node in our environment with a CPU utilization below 90%. The highest CPU utilization for this node is 73.3%, which is generally a great place to sit in terms of utilization, as it leaves room for CPU spikes. However, the average CPU utilization is generally lower than the maximum CPU utilization, and if this node is provisioned on a large instance type, like an EC2 m5.4xlarge, it’s likely over provisioned.
The response from the AI Assistant indicates that our node is operating efficiently. It also provides suggestions for potential optimizations, important attributes about the node, and links to Kubernetes charts and dashboards for continued monitoring:
Wrap up
To summarize the use cases explored in this post, we’ve used the Splunk AI Assistant to audit compliance and cost by identifying expensive or underutilized infrastructure and discovering service configurations that may be out of compliance.
In our next post, we’ll use the AI Assistant to explain specific metrics, contextualize data, and provide feedback.
Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!
Resources
- Splunk Observability Cloud’s AI Assistant Series: Analyzing and Troubleshooting in Real-Time
- AI Assistant in Observability Cloud
- Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability
- Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Compliance and Cost Control YouTu...
- Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability...
- Splunk Observability Hub
