Although it might seem daunting, as we’ve seen in this series, manual instrumentation can be straightforward and a powerful tool in a complete observability practice. In this series, we dig into manual instrumentation to see how we can use it to easily build out observability from scratch. In our first post, Manual Instrumentation with Splunk Observability Cloud: The What and Why, we learned about what manual instrumentation is and why you need it. Our second post went through the process of manually instrumenting the backend of a real, un-instrumented full-stack application, commit by commit. In this post, we will look at our un-instrumented frontend and go step-by-step through the process of manually instrumenting it.  Example Application Overview Recap GitHub repository: https://github.com/splunk/evangelism-public/tree/main/manual-instrumentation To review, our example application is called Worms in Space. It consists of a React frontend with an Elixir backend and provides a full GraphQL API. Worms in Space allows customers (who happen to be worms) to schedule spacewalks in… space.  Space worms can use the UI or GraphQL API to: View available time slots Schedule space walks during predefined or custom time slots View all scheduled space walks Cancel space walks Here’s what the frontend UI looks like:  And here you can interact with the GraphQL API via GraphiQL:  If you'd like to follow along on GitHub commit-by-commit, check out the GitHub repository, otherwise, you can read on to follow each commit here. Why Splunk RUM for Frontend Instrumentation? Splunk RUM (Real User Monitoring) within Splunk Observability Cloud is designed to monitor the performance and user experience of web and mobile applications from the perspective of actual end-users. It collects data directly from the user's browser or mobile device, and provides insights into how real users interact with and experience your application for complete visibility. When instrumenting a frontend application, it’s important to work with a solution that provides: Support for multiple browsers and mobile applications Complete instrumentation of full user sessions without degrading performance or user experience Out-of-the-box, real-time metrics, dashboards, and charts around performance, user interactions, user sessions, and page loads AI-driven analytics to identify performance improvements across every page, resource, component, and third-party dependency Splunk RUM gives us all these things and more – plus it can be implemented in just a few commits. Let’s dig into the implementation.  Frontend Instrumentation with Splunk RUM Step 1: Add Splunk RUM Dependency Git commit: 9d23d770 For full documentation on the Splunk RUM installation process, check out the installation docs. We’re going to install Splunk RUM via NPM, so the first thing we need to do within our frontend application is add the Splunk RUM dependency to our package.json file: @splunk/otel-web is the Splunk Distribution of OpenTelemetry for JavaScript and abstracts away many of the underlying OpenTelemetry and Splunk-specific details.    Step 2: Configure Splunk RUM Git commit: e0caea8a Next, since our application uses Typescript, we’ll configure Splunk RUM in a assets/js/rum.ts file: This configuration sets the: realm: Your Splunk Observability Cloud realm (us0, us1, eu0, etc.). You can find your realm in your Splunk Observability Cloud organization settings. The SDK automatically constructs the correct ingest endpoint from the realm. rumAccessToken: Your RUM access token. This is different from APM tokens and has limited, frontend-safe permissions. You can generate a RUM access token by navigating to Settings → Access Tokens in Splunk Observability Cloud. applicationName: Your application name as it will appear in RUM dashboards. Choose something descriptive that distinguishes this frontend from other services. deploymentEnvironment: Deployment environment (development, staging, production). This enables filtering in RUM views. Security note: RUM tokens are designed to be exposed in frontend code. They only have permission to send RUM data and cannot access other Splunk Observability Cloud resources.  Step 3: Initialize Splunk RUM in React Application Entry Point Git commit: d65649fe We initialize RUM in our application entry point in our assets/js/app.tsx before React starts. You’ll want to do this as early as possible in the application source: The installation documentation states to import or require the splunk-instrumentation.js file, or in our case, our assets/js/app.tsx file, before other files to ensure that the instrumentation runs before the application code. With this single initialization, Splunk RUM will now automatically instrument: Performance Metrics Network Activity User Interactions JavaScript Errors For a complete list of data collected by the RUM Browser agent, check out the RUM Browser data model docs.  Step 4: Add Custom Instrumentation for Business Logic Git commit: ffaec857 While automatic instrumentation covers browser metrics, we add in our manual instrumentation with custom spans to track business-critical operations that are unique to our application. In this example, this critical operation is actually scheduling a spacewalk: This is just a snippet from our AstronautTimeSlots.tsx file. For the complete implementation, including custom spans for delete operations, and delete operations, see the full source code on GitHub. These custom spans appear in RUM traces alongside automatic instrumentation, providing complete visibility into user workflows. See custom instrumentation documentation for more patterns.  Let’s now see what this manual instrumentation looks like in Splunk Observability Cloud.  What You'll See in Splunk Observability Cloud With this instrumentation in place in just three commits, we immediately gain visibility into: Key performance metrics like: user sessions, page views, JavaScript errors, and core web vitals  Browser and device analytics: device type breakdown and performance comparison Detailed session breakdowns showing individual user journeys with timelines, waterfall views, errors, and replays End-to-end trace views showing frontend and backend correlation Out-of-the-box and customizable dashboards, and alerts Now that our application is instrumented, we can go into Splunk Observability Cloud, navigate to Splunk RUM to see that our worms-in-space-frontend application is now sending RUM data: Here we can quickly see those key performance networks like page views, JavaScript errors, and web vitals.  If we click into our worms-in-space-frontend from this view we can see a more detailed, filterable view of page views, page loads, and interactions: And then selecting the specific interaction, we can go into the waterfall view to see the detailed trace breakdown of user interactions and network requests: In this waterfall breakdown, we can see custom spans that we manually instrumented like our delete_scheduled_walk. We can select this span to explore a more detailed view with the complete span tags that we manually added in our AstronautTimeSlots.tsx file:  Manual instrumentation success via Splunk RUM 🎉. Wrap Up In just three commits, we've transformed our un-instrumented React frontend into a fully observable service with Splunk RUM. The combination of automatic instrumentation provided out-of-the-box and the manual instrumentation we added for our specific business logic provides complete visibility into our frontend application performance. This level of observability transforms mysterious production issues into clearly diagnosed problems with obvious solutions so space worms can seamlessly schedule and reschedule their spacewalks at any time. Join us in our next post to follow along, step-by-step, as we update our Worms in Space application to use the OpenTelemetry Collector.   Ready to manually instrument your own frontend? Start with Splunk Real User Monitoring, and use Splunk Observability Cloud's 14-day free trial to easily visualize your telemetry data in one unified platform. Resources Manual Instrumentation with Splunk Observability Cloud: The What and Why Manual Instrumentation with Splunk Observability Cloud: How to Instrument Backend Applications Splunk Observability Cloud Overview ... View more

Manual instrumentation is an invaluable observability tool. In this series, we dig into manual instrumentation to see how we can use it to easily build out observability from scratch. In our first post, Manual Instrumentation with Splunk Observability Cloud: The What and Why, we learned about what manual instrumentation is and why you need it. In this post, we'll look at a real, un-instrumented full-stack application and go step-by-step through the process of manually instrumenting its backend, commit by commit. Preface: SDK-based Automatic Instrumentation Before we jump in, we want to call out that while we will be using manual instrumentation to create custom telemetry data, we’ll also be using OpenTelemetry’s SDK-based automatic instrumentation to hook into our web framework, HTTP requests, and database so we don’t have to manually create spans for framework operations.  A true zero-code example of automatic instrumentation would require no code changes, but since zero-code auto-instrumentation doesn’t exist (yet) for our application’s Elixir backend, we’ll use SDK-based automatic instrumentation for framework operations and full manual instrumentation for our business logic instrumentation.  Example Application Overview  GitHub repository: https://github.com/splunk/evangelism-public/tree/main/manual-instrumentation Our un-instrumented example application is called Worms in Space. It consists of a React frontend with an Elixir backend and provides a full GraphQL API. Worms in Space allows customers (who happen to be worms) to schedule spacewalks in… space.  Space worms can use the UI or GraphQL API to: View available time slots Schedule space walks during predefined or custom time slots View all scheduled space walks Cancel space walks Here’s what the frontend UI looks like:  And here you can interact with the GraphQL API via GraphiQL:  If you'd like to follow along on GitHub commit-by-commit, check out the GitHub repository, otherwise, you can read on to follow each commit here. Backend Instrumentation Implementation Again, our example application is an Elixir backend with a GraphQL API for scheduling spacewalks. Even if you aren’t using Elixir or GraphQL, you can still use the following general steps as a guide to instrument your own backend application: Add the necessary OpenTelemetry dependencies Configure OpenTelemetry Initialize instrumentation Add custom telemetry data Let's walk through each of these steps commit-by-commit to instrument our Worms in Space backend.  Step 1: Add OpenTelemetry Dependencies Git commit: 5f926245 The first thing we need to do within our backend application is add the OpenTelemetry dependencies to our dependency file, in our case our mix.exs file: The list of dependencies can be found in the OpenTelemetry Language APIs & SDKs doc under the Elixir/Erlang dependencies section. Since we’re using the core OpenTelemetry SDK, we’ll need to include OpenTelemetry, the OpenTelemetry API, and the OpenTelemetry exporter. Our Elixir application is using Phoenix for HTTP requests, so to auto-instrument our controllers, plugs, and requests, we’ll need to include the OpenTelemetry dependency for Phoenix. Phoenix uses Cowboy as its default HTTP server, so we’ll also include the OpenTelemetry Cowboy dependency so we can low-level HTTP requests. Our application uses Ecto for database interactions, so we’ll include the OpenTelemetry Ecto dependency, and then  GraphQL operations in Elixir are resolved via Absinthe, so we’ll need the OpenTelemetry Absinthe dependency, as well. Finally, the OTLP exporter is required for sending telemetry data to Splunk Observability Cloud.  Step 2: Configure OpenTelemetry SDK and OTLP Exporter Git commit: e45ee707 Next, we configure the OpenTelemetry SDK in our config/runtime.exs file: This configuration establishes our service identity, sets up the batch processor for efficient trace export, and configures the OTLP exporter to send traces directly to Splunk Observability Cloud. Note: in this current configuration, we are directly sending out telemetry data to our Splunk Observability Cloud backend. In most cases, you would want to set up an OpenTelemetry Collector to collect, process, and export telemetry data to your observability backend. In a subsequent post, we’ll look at how we can update our current configuration to use the OpenTelemetry Collector for increased resilience, flexibility, and processing capabilities. Step 3: Initialize Instrumentation Components in Application Git commit: f7113fec We next initialize the automatic instrumentations in our application.ex startup code: Notes on this setup: Cowboy Setup initializes the foundational HTTP server instrumentation Phoenix Adapter setup ensures correct integration with Phoenix's HTTP server Setup order matters. Initialize instrumentation libraries before your application starts accepting requests Include logging to help verify configuration at startup This setup initializes the dependencies referenced in the first commit in Step 1 and provides immediate visibility into HTTP requests, database queries, and GraphQL operations without any code changes. Every HTTP endpoint, database query, and resolver will automatically generate traces. Step 4: Add Custom Spans for Business Logic Git commit: 9278ef62 While SDK-based automatic instrumentation covers the framework layers, we need to add custom spans to capture our application’s unique business logic. We do this by setting attributes on specific spans within our create function heads in our resolvers.ex file so that every time a spacewalk is created, custom telemetry data with specified attributes is exported to Splunk Observability Cloud: Note: I’ve collapsed this method for brevity, but the full method and file can be viewed on GitHub in the manual-instrumentation/worms_in_space/lib/worms_in_space_web/api/resolvers/resolvers.ex file. These custom spans provide deep visibility into our spacewalk scheduling business logic, including success/failure rates, scheduling types, and performance metrics for critical operations. Some of the most common attributes to include in manual instrumentation are things like customer level or user ID and can be configured based on your unique business use-cases. Environment Configuration Finally, we set our environment variables for deployment flexibility: This configuration allows seamless deployment across development, staging, and production environments while maintaining complete trace context and correlation. What You'll See in Splunk Observability Cloud With this instrumentation in place in just four commits, we immediately gain visibility into: Complete request flows from HTTP request to database query with timing and status GraphQL operation performance with query names and execution time   Database query analysis including SQL statements and execution time Custom business metrics like the ones we added around spacewalks Error tracking with full stack traces and contextual attributes Going into Splunk Observability Cloud, we can visit the service overview of our Worms in Space backend in Splunk Application Performance Monitoring. Here we can see valuable service metrics, Service Map, Tag Spotlight, errors, and traces:  If we open the traces from this view, we can see long running traces: We can open one of those long running traces, and then we can select a span to view the custom attributes that we manually instrumented earlier:  Manual instrumentation success 🎉. Wrap Up In just four commits, we've transformed our un-instrumented Elixir backend into a fully observable service. The combination of automatic instrumentation for framework operations and manual instrumentation via custom spans for business logic provides complete visibility into application performance and behavior. When issues arise at 2 AM, we’ll now know exactly which spacewalk scheduling operation failed, how long database queries are taking, and whether the problem is in our business logic or our infrastructure. This level of observability transforms mysterious production issues into clearly diagnosed problems with obvious solutions so space worms can seamlessly schedule their spacewalks at any time. Join us in our next post to follow along, step-by-step, as we manually instrument the React-powered frontend of our Worms in Space application.  Ready to add context to your own backend apps? Start with OpenTelemetry's documentation on Language APIs and SDKs, and use Splunk Observability Cloud's 14-day free trial to easily visualize your telemetry data in one unified backend platform. Resources Manual Instrumentation with Splunk Observability Cloud: The What and Why Splunk Observability Cloud Overview OpenTelemetry Language APIs & SDKs         ... View more

If you've ever worked with distributed systems, you’ve likely felt the pain of a frontend throwing errors, while your backend is processing requests slower than usual, and somewhere between your resolvers and database queries something unknown is going wrong. But what are the specific issues, where exactly are they occurring, and why did they happen in the first place? This is exactly the kind of insight observability provides, and the benefits of having such visibility into systems are endless. Faster insight into problems means proactive prevention of disruptions and performance issues. Visibility into the root cause of issues leads to quick pinpointing of the exact lines of code that need to be fixed, which means the impact to our users is minimal or non-existent, and our customers stay happy.   Observability platforms make it easy to gain visibility into application ecosystems through unified, intuitive UIs, and automatic instrumentation. Collecting and exporting the data required for this insight is often automatic through tools like automatic discovery and zero-code instrumentation. But even when automatic instrumentation is available, there are times when only you know what really matters for your application’s success. Adding in manual or custom instrumentation can provide the full picture when it comes to specific application logic or insight related to business-specific data. Manual instrumentation can be an invaluable tool in your observability toolkit, and in this series, we're going to dig into it to see how we can use it to easily build observability from scratch. We’ll look at a real application and take it from zero observability to fully instrumented, commit by commit. In this first post, we’ll learn about what manual instrumentation is and why you need it. Then, in subsequent posts, we'll look at a real, un-instrumented application, and go step-by-step through the process of manually instrumenting it.   What Is Manual Instrumentation?  Automatic instrumentation seems almost magical. Install an agent or library, restart your service, and poof -- metrics, logs, and traces start flowing into your observability platform. But what about the parts of your code that don’t get picked up automatically? The times where your services aren’t written in automatic instrumentation-supported languages? Or the times when you need more context than what’s provided by auto-instrumentation? That’s where manual instrumentation comes in. Manual instrumentation is the process of explicitly adding observability code to applications to capture custom telemetry data. Observability platforms often provide auto-discovery, zero-code implementations – including the use of which was recently donated to the community by Splunk – that fully instrument systems. But not all languages and services are supported when it comes to auto-instrumentation. Even in cases where auto-instrumentation is possible, we sometimes need more detail. Custom instrumentation around business-specific telemetry data and specific code or custom attributes provides invaluable business context and captures the nuances for our unique applications. In these cases, manual instrumentation adds a ton of value.   Unlike with automatic instrumentation, with manual instrumentation, you take control and decide exactly what and how to instrument. This means that even if your observability platform doesn’t automatically provide visibility into your code base, and your application starts throwing errors at 2 AM, you’ll still be notified about the failures, and you'll know exactly why and where errors are occurring.  Why Manually Instrument? In cases where automatic-instrumentation for a language or service isn’t available or where out-of-the-box telemetry data isn’t capturing the full application picture, manual instrumentation is a necessity. Manual instrumentation offers full-stack observability, business logic insight, and custom metrics and attributes. Let’s look at each of these benefits. Full-Stack Visibility  Manual instrumentation helps complete the full application picture for complete visibility into performance and issues. With automatic instrumentation you can see things like degraded endpoint performance, but it won't always tell you which part of your custom sorting algorithm is causing bottlenecks. With manual instrumentation, you can add instrumentation around the specific code to identify exactly where time is being spent. Is the bottleneck in the data transformation? The third-party API call? Now you'll know. Manual instrumentation lets you add these detailed breadcrumbs throughout your code, capturing variable states, decision outcomes, and execution paths. It's like leaving yourself detailed notes about what your code was doing at each step. Business Logic Insight Your application isn't just a collection of HTTP endpoints and database calls. It's a carefully orchestrated system of business rules, calculations, and decision points. Manual instrumentation lets you trace through complex workflows like order processing, fraud detection algorithms, or multi-step approval processes. When something goes wrong with a customer's order at 2 AM, you want to know exactly which business rule failed and why. Manual instrumentation allows you to capture this business-specific insight. Custom Metrics and Attributes    Every application has its own set of unique and important metrics. Maybe it's the number of items in a shopping cart, the complexity score of a generated report, or the cache hit rate for your custom caching layer. Manual instrumentation lets you capture these application-specific metrics and add custom attributes that provide context, like customer tier, feature flags, or regional data.  Wrap up For cases where automatic instrumentation isn’t possible or doesn’t provide all the features you need, manual instrumentation gets you critical visibility to keep your applications up and running. When automatic instrumentation is available, it gets you to broad coverage quickly, but adding in manual instrumentation gets you deeper insights. Manual instrumentation helps you go beyond what’s possible with automatic instrumentation, so you have the power to see exactly what's happening, when it's happening, and why it's happening – turning mysterious production issues into clearly diagnosed problems with obvious solutions. Join us in our next post to see how you can easily go from zero observability to full visibility with manual instrumentation. Ready to take control of your observability today? Start with OpenTelemetry documentation and examples to begin instrumenting your application. If you need a platform where you can visualize all your telemetry data in one place, Splunk Observability Cloud's 14-day free trial is a great way to begin your observability journey. Resources Instrument back-end applications to send spans to Splunk APM Instrumentation methods for Splunk Observability Cloud Automatic discovery of apps and services OpenTelemetry Language APIs & SDKs   ... View more

If you’ve read our previous post on self-service observability, you already know what it is and why it matters. Self-service observability empowers teams to own their observability practice, eliminating cross-team dependencies and bottlenecks, and instead optimizes delivery and speeds up incident resolution. It also empowers your engineers to customize observability for their application’s unique needs. In this post, we’ll walk through how to implement self-service observability using Splunk Observability Cloud.  Start with Team Structure and Access Control To prevent teams from fumbling through an incident while sifting through an explosion of dashboards and charts, team structure and organization within an observability platform are critical. A successful self-service observability practice is organized and starts with team structure.  In Splunk Observability Cloud, Teams are a way to organize users into functional groups and efficiently connect them to the dashboards, charts, detectors, and alerts that matter to them. To create a Team:  Select Settings in the left navigation menu Go to Team Management  Select Create team, enter a name and description, and then Add members Save the Team by selecting Create team Team members will have access to a dedicated landing page that contains relevant information tailored to their needs. A landing page brings together team-specific: Dashboard groups Detectors triggered by team-linked alerts Favorited and mirrored dashboards (centralized dashboards that are editable across teams but customizable per team without affecting other mirrors) This setup helps everyone stay aligned and focused, whether during incidents or proactive monitoring.  For each team, team managers can be assigned to configure the proper roles and user management, including Admin, Power, Usage, and Read-only. This way, people only have access to the things they need.  Access controls can be set using Splunk Observability Cloud Enterprise Edition’s built-in RBAC by: Navigating to Settings and selecting Access Tokens Creating team-specific access tokens with the appropriate permissions Setting token usage limits to manage the maximum number of usage metrics This granular token provisioning protects your environment, enables usage tracking per team, and helps to proactively alert on usage thresholds so your teams can build and use the resources they need without unknowingly impacting cost.  Standardized Templates & Observability as Code Templating and automating the creation of observability resources increases the likelihood of adoption while also facilitating standardization through easily shareable resources. There are different ways to build templates, such as using the Splunk Observability Cloud API or the Splunk Terraform Provider. As long as your resources are in source control, there will be checks and balances, rollback capabilities, and higher developer velocity (because teams can focus on writing feature code rather than creating observability resources).  Here’s an example of how to use the Splunk Terraform provider to create a dashboard and chart in Splunk Observability Cloud:  Teams can also be specified from within the Terraform definitions so that dashboards show up within the specified Team’s landing page.  If you’d like to learn more about Observability as Code, check out our blog posts: Observability as Code and Let’s Talk Terraform.  Implement OpenTelemetry To enable self-service observability, you need to standardize the collection, processing, and export of telemetry data. OpenTelemetry provides this standardization through semantic conventions, including: Tagging conventions: service.name, deployment.environment, team Metric and span naming standards: http.server.duration Attribute requirements: cloud.provider, region, k8s.cluster.name When telemetry data is standardized, teams using observability resources can confidently find and understand their data, dashboards, and alerts. These resources are then reusable and familiar to users across teams, making it possible for incident responders to quickly jump in and provide assistance in emergencies. Using semantic conventions also helps your observability tooling aggregate across different cloud providers, frameworks, and programming languages to give a unified view of your business. Measure, Iterate, Expand Achieving Self-service observability is a journey. It is an incremental process that requires measuring adoption, identifying gaps, and iteration. Tracking adoption metrics, usage metrics, and cost can help you fine-tune your practice and deliver successful and complete visibility into your systems. We recommend you track some key metrics, below, to enable you to verify that teams are using observability, to understand and control costs, and to help justify your investment in observability. Adoption metrics to track:  Percentage of teams with at least one active detector Number of dashboards or detectors created per team Time from alert trigger to acknowledgement Support ticket volumes (this should decrease as your observability practice grows) Usage metrics to track:  Metric cardinality Time series volume Whether or not specific metrics are being used Cost optimizations: Filter unused or overly granular metrics Set rules for aggregation or dropping metrics Route data to reduce ingestion and storage costs Avoid Common Pitfalls Finally, when implementing a self-service observability practice, it’s essential to avoid common problem areas that can lead to chaos, increased cost, and operational silos. Below are some key pitfalls to watch out for, along with strategies to avoid them.  Don’t skip documentation Every observability resource should have a clear description, contact information for the owning team, and links to runbooks where applicable. It might sound extreme, but a runbook should accompany every detector. Here’s a Terraform example of a detector with proper documentation:  Don’t ignore costs With each team managing its observability, there is the potential for a telemetry data explosion or a rapid increase in the amount of telemetry data being exported to a backend observability platform. This explosion of data not only makes it difficult to process and analyze, but it also increases costs.  Within Splunk Observability Cloud, you can always get detailed insight to monitor and manage subscription usage and billing to ensure you stay within your defined limits:  In the Enterprise Edition of Splunk Observability Cloud, Metrics Pipeline Management (MPM) centralizes a solution for managing metrics and helps you easily identify metric usage in-product. You can then adjust for metric explosion in-product or adjust the metric in code, and even adjust storage settings to lower costs and improve monitoring performance.  Don’t create silos Giving every team control over their observability practice can quickly dissolve into chaos if standards aren’t in alignment. Avoid silos and promote a unified strategy by:  Using shared naming conventions Providing shared templates Promoting best practices through enablement (office hours, onboarding sessions, communication channels, internal demos, lunch-and-learns) Always encourage cross-team collaboration and create opportunities for shared learning to ensure your self-service observability practice grows and thrives.  Wrap Up With an organized team structure, consistent telemetry powered by OpenTelemetry, and observability as code, your teams will not only be able to move faster, but they will also be empowered with the insights they need to respond to issues with greater efficiency and confidence.  Ready to take the first step in your self-service observability journey? Start building your self-service observability practice today with Splunk Observability Cloud’s 14-day free trial.  Resources Self-Service Observability: How to Scale Observability Adoption Through Self-Service Building a Self-Service Observability Practice Let’s Talk Terraform Introduction to the Splunk Terraform Provider Observability as Code Enable self-service observability ... View more

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. Specifically, this series goes into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start at the beginning of this series, check out: Identifying Unknown Unknowns. In this sixth post, we’ll explore how the Splunk AI Assistant can be used to generate scripts that serve as a launchpad for automation code using the Splunk Observability Cloud API.  Generating Scripts with the Splunk AI Assistant As an engineer or administrator of Splunk Observability Cloud, I may want to automate tasks related to observability resources using the Splunk AI Assistant and the Splunk Observability Cloud API.  The AI Assistant is pretty flexible in terms of the types of questions you prompt it with, as long as the questions are related to the code integrated with Splunk Observability Cloud and the data within it. We can ask the AI Assistant to generate code that interacts with the Splunk Observability Cloud API. For example, I can ask the AI Assistant to write a Python script that adds my user to a team in Splunk Observability Cloud – all that I will need is my user ID, and the team ID I want to add my user to.  To grab my user ID, I’ll navigate to my profile in Splunk Observability Cloud and copy and save my user ID:  And then I’ll navigate to the team and grab its ID:  Note: you’ll notice my user, Caitlin Halla, is not currently a member of this team. Using my user ID and this team ID, we can then prompt the AI Assistant to write a Python script that adds me, user GQSnQ4TAwAg, to team Fuqe6XzA4AA using the Splunk Observability Cloud API. The response we get from this prompt includes the code we’ll need for our Python script:  We can then copy and paste this code into our code editor and replace the placeholders and required values, like our access token (with administrative access), with real values:  After running our script, if we navigate back to the team in Splunk Observability Cloud, we can see that my user was successfully added to this team:  Wrap up To summarize what we explored in this post, we learned how the Splunk AI Assistant can help us generate a Python script to automate specific tasks, like adding a Splunk Observability Cloud user to a specific team. This type of interaction with the Splunk AI Assistant highlights the ability to use the Assistant for Observability as Code. The AI Assistant can provide you with code that interacts with the Splunk Observability Cloud API and draft SignalFlow programs that you can then insert into charts. It can even generate the charts via Python scripts.  With the Splunk AI Assistant, observability possibilities are continually expanding. In this series, we’ve discussed some of the use cases currently available within the AI Assistant, but these will only evolve and get better with time. As features expand, we look forward to seeing you in a future AI Assistant series. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Previous post in the Splunk Observability Cloud’s AI Assistant Series: Onboarding New Hires or New Users of Splunk Observability Cloud AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Observability as Code YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the fifth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. In this series, we go into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start at the beginning of this series, check out our first post: Identifying Unknown Unknowns. In this fifth post, we’ll explore how the Splunk AI Assistant can help quickly onboard new hires and/or help those who might be new to Splunk Observability Cloud.  Onboarding New Hires Deep contextual, interpreted, and real-time data isn’t just helpful to experienced engineers - as we saw in our last post, Explaining Metrics and Providing Feedback. Imagine you’re an engineer who has just joined a new company. You’re trying to understand the services and environment your team owns and how everything works together. You can use the AI Assistant to get an understanding of the environment and the data that’s flowing between services. For example, you might want to know what kind of data is being sent to the Payment service in your team’s Online Boutique environment:  From this prompt, the AI Assistant uses log data to identify what’s happening in the Payment service and returns the data types being sent and processed. In the response, we can see details on how the Payment service is processing payment transactions, how HTTP requests are made to the Payment service, order confirmation integration with an email service, and retry mechanisms for payment processing:  Finally, we have a nice summary of what the Payment service does:  This kind of information is super powerful for anyone who is new to the organization and trying to quickly get up to speed on how service architecture and how they work together. Documentation is helpful during onboarding, but the AI Assistant provides real-time, up-to-date information quickly and interactively.    Wrap up To summarize the use case explored in this post, we’ve used the Splunk AI Assistant to onboard new hires and build familiarity with Splunk Observability Cloud. In our next post, we’ll learn how to leverage the AI Assistant for Observability as Code by generating scripts that can interact with the Splunk Observability Cloud API. Scripts like these can be expanded to create dashboards, charts, detectors, and more. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Previous post in the Splunk Observability Cloud’s AI Assistant Series: Explaining metrics and providing feedback AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Onboarding New Hires and New Users to Splunk Observability Cloud  YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. In this series, we go into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start with how to access the Splunk AI Assistant and identify unknown unknowns within a service environment, check out the first post in this series: Identifying Unknown Unknowns. In our second post, Analyzing and Troubleshooting in Real-Time, you can see how to use the AI Assistant to analyze trace errors and database query performance and gain insights into detectors and alerts. Our third post digs into how to leverage the Splunk AI Assistant to ensure compliance and optimize cost requirements. In this fourth post, we’ll explore how the Splunk AI Assistant can help explain unfamiliar or custom metrics and how it can provide real-time, contextual feedback to help analyze performance, identify optimizations, and troubleshoot faster for a reduced Mean Time to Resolution (MTTR).  Splunk AI Assistant Explains Metrics Within a company’s engineering organization, different teams may implement their own custom metrics specific to the services they own and manage. When troubleshooting an incident that spans services and teams, navigating these custom metrics and their meanings can slow down investigation and MTTR. Not only that, but engineers often utilize third-party tools that can be instrumented with custom metrics that don’t conform to internal standards or conventions. Interpreting these metrics can have a similar negative impact on troubleshooting time and MTTR. Let’s look at an example.  Our engineering team utilizes the data structure server, Redis, but I haven’t spent a lot of time working with Redis, so I’m unfamiliar with all of the metrics associated with our Redis instance. I’m on-call and dealing with an active incident, and I’m seeing a lot of Redis metrics pop up around cache hit rate percentage. I’m not sure what this means, but I can use the Splunk AI Assistant to gain more context and troubleshoot the issue.  From within Splunk Observability Cloud, I’ll navigate to Infrastructure and then select Datastores. I’ll scroll down to Redis and select Redis instances to get a detailed overview of my Redis instances:      From within the Redis instance view, I can open the AI Assistant and ask detailed questions about specific instance metrics, like this cache hit rate percentage. I can also ask it for contextual data to help me determine if the metric data is good or bad.  In my prompt, I’m going to specify the instance I’m investigating because, as you’ll remember from our previous post, Analyzing and Troubleshooting in Real-Time, the more context you provide to the AI Assistant, the more complete responses will be:  I’ve asked the AI Assistant to explain what the cache hit rate percentage metric is and how I can interpret it. The response returns analysis for this specific metric, showing that over the last four hours, the hit rate was 67%.  The interpretation of the contextual data in the response suggests that this is a moderate hit rate, indicating there is room for optimization. More details on suggestions for further optimization include reviewing the cache configuration, analyzing miss patterns, and application logic:  The response also provides important relevant Infrastructure attributes along with relevant dashboards and charts, complete with hyperlinks for further insight:  Wrap up To summarize the use case explored in this post, we’ve utilized the Splunk AI Assistant to explain unfamiliar metrics and provide real-time, contextual feedback, enabling us to troubleshoot quickly and reduce MTTR. In our next post, we’ll use the AI Assistant to streamline the onboarding process for new hires or new users of Splunk Observability Cloud to get everyone up to speed fast. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Previous post in the Splunk Observability Cloud’s AI Assistant Series: Auditing Compliance and Cost AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Explaining Metrics and Providing Feedback YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. Specifically, this series goes into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start with how to access the Splunk AI Assistant and identify unknown unknowns within a service environment, check out the first post in this series: Identifying Unknown Unknowns. In our second post, Analyzing and Troubleshooting in Real-Time, you can see how to use the AI Assistant to analyze trace errors and database query performance and gain insights into detectors and alerts. In this third post, we’ll learn how to leverage the Splunk AI Assistant to ensure we remain in compliance with our organization’s specific requirements. We’ll also see how the AI Assistant can analyze our infrastructure and identify areas that may be underutilized, which in turn will help us reduce costs. Compliance Organizations often implement code-level tagging requirements for compliance, security, cost, and operations. For example, it’s a requirement that all of the services in our Online Boutique environment set a tag value for the attribute tenant.level to easily distinguish between service tiers. This tenant.level attribute has three tag values – gold, bronze, and silver – as seen here in Tag Spotlight within Splunk Application Performance Monitoring:  We can use the AI Assistant to audit the data in our environment and ensure we are in compliance with this specific organizational requirement. We’ll ask the AI Assistant if any services in the Online Boutique environment aren’t using this required tenant.level tag. In the response, we can see that our Ad service is missing the required tenant.level tag:  Note: In the AI Assistant pane, if you have “Use current page filters” toggled on and you try to run this query from a specific page, like the paymentservice detailed view, you might not see the full results you expect. Querying from this page would set the context to just that of the paymentservice. If we want to interrogate our entire environment and all services instead, we would need to move up a level and execute the query from our services view that lists and sets the context as all services in our environment.  If we go to our Ad service in Splunk APM, we can verify that the data from it is in fact missing the required tenant.level tag: Cost Control Similar to what we saw in our last post about reducing alert noise and ecosystem hygiene, we can use the AI Assistant to analyze our infrastructure and help us identify areas that might not be efficiently utilized. This can help us capacity plan and reduce costs when allocating infrastructure.  We’ll have the AI Assistant analyze our Kubernetes infrastructure by prompting it to tell us whether it’s underutilized and identify nodes in Splunk Infrastructure Monitoring with all-time high CPU usage below 90% in our Online Boutique environment. The AI Assistant’s analysis identifies a node in our environment with a CPU utilization below 90%. The highest CPU utilization for this node is 73.3%, which is generally a great place to sit in terms of utilization, as it leaves room for CPU spikes. However, the average CPU utilization is generally lower than the maximum CPU utilization, and if this node is provisioned on a large instance type, like an EC2 m5.4xlarge, it’s likely over provisioned.  The response from the AI Assistant indicates that our node is operating efficiently. It also provides suggestions for potential optimizations, important attributes about the node, and links to Kubernetes charts and dashboards for continued monitoring:  Wrap up To summarize the use cases explored in this post, we’ve used the Splunk AI Assistant to audit compliance and cost by identifying expensive or underutilized infrastructure and discovering service configurations that may be out of compliance. In our next post, we’ll use the AI Assistant to explain specific metrics, contextualize data, and provide feedback. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Splunk Observability Cloud’s AI Assistant Series: Analyzing and Troubleshooting in Real-Time  AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Compliance and Cost Control YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at how to use the Splunk AI Assistant by digging into some practical, real-world, real-time examples. This series explores the specific use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics or providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code In our first post in this series, we saw how to access the Splunk AI Assistant and then explored how we could identify unknown unknowns within our service environment. The AI Assistant identified that our latest release could be causing high error rates and high latency. Before jumping into this second post, check out that first post: Identifying Unknown Unknowns.  This second post picks up where we left off and jumps into using the Splunk AI Assistant to: Perform deploy and release checks Investigate database query performance Interrogate active detectors and alerts Deploy and Release Checks We can use the AI Assistant to compare new code releases to previous versions. Let’s see this in action by asking the AI Assistant to compare the current latency of our Payment service to the prior version. We’ll ask it to compare Payment service performance for the current and previous versions in our Online Boutique environment and identify changes in behavior or error patterns for the latest release: The AI Assistant returns a comparison that shows the latest version of our service, version 350.10, has a much higher latency than the previous version, version 350.9:  Scrolling down to the error rate analysis, we can see version 350.10 has an error rate of over 30,000 errors, while version 350.9 has 0 errors. The analysis also shows that the common error for the latest version is “Invalid request with HTTP status code 401”: The AI Assistant also highlights observations and recommendations for further investigation and resolution of the issue:  The Observations section shows that the latency increase and error rate surge are significant and indicate “potential issues with the latest release.” The Recommendations include ideas for further investigation, analysis of code changes between versions, and rolling back to version 350.9 to maintain service stability. Analyze Trace Errors While investigating real-time issues, looking at specific trace data is helpful to get a deeper understanding of the root cause. We can use the AI Assistant to analyze trace errors for this troubleshooting in real-time use case. Here, we ask the AI Assistant to analyze trace errors specifically for our Payment service in our Online Boutique environment over the last 15 minutes: It’s important to note that being more specific with your AI Assistant queries – for example, specifying services, environments, time windows, and data types will help the assistant return more specific and useful responses.  In the response for our current prompt, we can see a list of trace errors for the Payment service with dynamic links to each of the traces: We can then ask for a deeper analysis of traces by prompting the AI Assistant with a specific trace ID:    The AI Assistant identifies the root cause of the errors as a 401 status code or unauthorized access to one of our third-party APIs: the ButtercupPayments service. The response includes the error message and error tags for the invalid request, and also an analysis of trace performance:  Scrolling down in the response, we can see the Performance Issues section that provides helpful information about the duration of the spans within our specific trace. The duration for this trace was relatively short, which hints at a pretty immediate rejection of the request. This section also highlights the service interaction, which explains how the data is requested. In this case, our Payment service is trying to request the Buttercup Payments API and failing due to this authorization issue: Looking at the recommendations, the AI Assistant suggests reviewing the authentication mechanisms for the Buttercup Payments API to ensure the credentials or tokens are valid, along with a couple of other recommendations:  Selecting the hyperlink at the end of this response will take us to the trace in Splunk APM so we can look at the data in detail and explore further: Database Query Performance Not only can we ask the AI Assistant to analyze traces and metrics related to our services, but we can also ask it to analyze database performance and investigate database queries. We can ask detailed questions like, “What are the top 5 worst performing database queries in the online boutique environment?” In the response to this question, we get a list of the database queries that have high latency, along with the number of times those queries were executed:  Detector and Alert Insights The Splunk AI Assistant can also provide insight into detectors and alerts. If we navigate to Detectors & SLOs in Splunk Observability Cloud and select critical alerts, we can see we have an active high error rate alert for our Payment service:  If we select that alert and scroll down in the alert window, we can get the Incident ID of this specific alert:  We can copy that Incident ID and then use that ID to ask the AI Assistant to explain this specific alert and tell us why it was triggered. In the response, the AI Assistant provides a summary of the alert details along with an explanation of the triggering conditions and why they were triggered:  We started investigating this specific alert by navigating to the Detectors & SLOs page view in Splunk Observability Cloud, but we could have also queried the AI Assistant to get information about active or frequently triggered detectors. The response to this query guides where to focus our attention: If these detectors are frequently triggered, it could result in service performance degradation and impact customer experience. The response to queries like this can also help us easily identify alerts that are triggered too frequently and are not actionable, which helps reduce alert noise.  Wrap up To summarize, this second post in our Splunk Observability Cloud’s AI Assistant in Action series focused on using the AI Assistant to perform deploy and release checks, analyze trace errors, investigate database query performance, and interrogate active detectors and alerts. This enabled quick insight into root cause analysis, service performance, and deployment impacts for quick issue resolution.  Stay tuned for our next post, in which we’ll use the AI Assistant to help manage organizational compliance and infrastructure costs. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Splunk Observability Cloud’s AI Assistant Series: Identifying Unknown Unknowns AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Deploy and Release Checks YouTube video Splunk Observability Hub ... View more

Agentic AI powers the Splunk AI Assistant within the Splunk Observability Cloud interface to help you quickly and easily tap into the health of your applications and infrastructure. Simply asking the AI Assistant your observability questions using natural language does the hard work of querying your environment and returning the relevant information you need to find root cause, triage, troubleshoot, and reduce Mean Time To Resolution (MTTR). The AI Assistant can even generate the SignalFlow you need to create custom charts and dashboards.  In this Splunk Observability Cloud’s AI Assistant in Action series, we will look at how to use the Splunk AI Assistant by digging into some practical, real-world, real-time examples. We’ll explore the specific use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining or providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code This first post will explain how to access the Splunk AI Assistant, and then we’ll learn how to use it to identify unknown unknowns within our service environment.  Accessing the Splunk AI Assistant You can access the Splunk AI Assistant from Splunk Observability Cloud by selecting the AI Assistant icon located in the upper right corner of the screen: Selecting this icon opens the interactive window to chat with the AI Assistant. Once the AI Assistant pane is open, you’ll see pre-populated prompts that are a great starting point if you’re new to the AI Assistant: Here’s an example response for the first prompt, "What can you help me with?”: As you can see, the AI Assistant can access metrics, traces, and logs in Splunk Observability Cloud. It can also help you analyze detectors, alerts, incidents, and draft SignalFlow.  Identifying Unknown Unknowns As an engineer, a great way to start the work day is to interrogate the AI Assistant with a question like, “What should I know about my environment?” Whether you’re back in the office after a weekend away, jumping into an on-call rotation, or simply catching up on a recent incident to help troubleshoot, using the AI Assistant to gain quick insight into unknown unknowns is a powerful resource.  Let’s ask a question about one of our environments. We’ll prompt the AI Assistant to tell us something we might not be aware of for one of the service environments we own, the Online Boutique. We’ll ask it to point out interesting or concerning things related to this environment: Looking at the response, we can see the AI Assistant performs analysis and then identifies that the Payment service has a high error rate. It also identifies common error tags like HTTP 401 status codes and high latency concerns: We can also see that the AI Assistant analyzed the upstream and downstream dependencies of the services in our environment: It also made some recommendations based on its findings. It hyperlinked to the services involved, so we can easily investigate those services in detail by moving over to Splunk Application Performance Monitoring for further troubleshooting. In this case, we’re hyperlinked to our Payment service and can explore real-time issues directly in Splunk APM:  My team owns the Payment service, and I know that code changes were recently deployed. A great next step would be to use the AI Assistant to interrogate our system and investigate if this latest release could be causing these high error rates and latency.  Wrap up To summarize, this first post in our Splunk Observability Cloud’s AI Assistant in Action series focused on using the Splunk AI Assistant to quickly identify unknown unknowns within our environment.  Stay tuned for our next post, in which we’ll use the AI Assistant to analyze and troubleshoot in real time, investigating error rates and latency and comparing deployments and releases. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Identifying Unknown Unknowns YouTube video Splunk Observability Hub ... View more

Web applications unfortunately present a target rich environment for security vulnerabilities and attacks. Adding in sometimes hundreds of third party dependencies and multiple services coordinating across a variety of technologies exponentially increases the impact and likelihood of these threats. Thankfully, Splunk AppDynamics (AppD) reduces the risk of application security exposure and additionally provides many features and capabilities to optimize hybrid and on-prem application performance. To name a few, AppD offers: Features to help identify application issues in real-time and tools to then prioritize how to tackle those issues based on business metrics and impacts  Root cause analysis from anywhere in your stack with AI-driven anomaly detection Business iQ Business performance monitoring to understand how application performance directly influences critical business metrics Digital Experience Monitoring (DEM) complete with Real User Monitoring (RUM) and Synthetic monitoring for insight into how end-users interact with our applications and observe availability and performance of web apps and APIs Network monitoring to quickly isolate network performance issues with complete visibility into owned and unowned network, ISP, API, SaaS, and third-party services Application security from inside your runtime environment with the detection of vulnerabilities, blocking of attacks, and insight into potential business impacts  In this post, we’re going to dig into the capabilities provided by Splunk AppDynamics with Cisco Secure Application and check out how they help enhance application security with real-time threat detection and prevention.  Splunk AppDynamics with Cisco Application Security With Cisco Secure Application, you can easily get complete security insight and protect your applications at runtime. Typically, security scans take place before an application is deployed to production, and scans continue to take place on a monthly or quarterly basis. But Cisco Secure Application continuously detects and blocks attacks. It also reveals the potential business impacts of these attacks and vulnerabilities so you can see and fix what matters to minimize business risk and easily prioritize issues.  Cisco Secure Application features are built into Splunk AppDynamics APM Agents for:  Java .NET Core Node.js Note: to monitor the application security, you must first enable security for the application using the Cisco Secure Application dashboard.  Overview The Cisco Secure Application home page provides a comprehensive and quick view of attacks and vulnerabilities.  Business transactions provide details about the vulnerability risk of a business transaction and the overview of vulnerabilities, attacks, and applications helps you quickly identify and respond to security incidents.  Applications The Applications page provides details on monitored nodes that are registered with Cisco Secure Application for the managed applications. Here, you’ll find a centralized view of all security-related details for your applications. This real-time data allows you to detect and respond to any applications that might not have security enabled for complete application coverage.  Stats on the nodes of managed applications are also available in this view. Nodes are categorized as: Active: the number of nodes that are actively communicating with the AppDynamics Controller Supported: the number registered with the Cisco Secure Application  Ready: the number of nodes that can be successfully registered Enabled: the total number of nodes that have security enabled for the applications Secured: the number of secured nodes Trend: the number of supported, enabled, secured, active, and ready nodes against the day of the month You can also quickly navigate to the flow map by clicking on the map icon to get a holistic view of the overall application performance.  Libraries The Libraries page breaks down details for existing libraries used within applications that require remediation.  Applications that use the corresponding library along with Tier (the application tier that is vulnerable because of its relationship to the corresponding library) are also easily visible. You can again click the map icon next to an application to view the application flow map in the AppDynamics dashboard.  This view also provides quick insight into:  The Highest Cisco Security Risk Score – an estimate of exploitation based on real-time events of the vulnerability in the wild A score based on the Highest Common Vulnerability Scoring System (CVSS)  Total vulnerabilities or the number of vulnerabilities based on severity  Remediation – the recommended version of the library that can be used for remediation Status of the vulnerable libraries Libraries that are categorized as "Critical" are potentially exploitable by a bad actor, which could disrupt your services or worse give access to customer data. These items should be prioritized for immediate triage. You can also click on a specific library to view vulnerabilities for that library:  Having this information easily visible means teams can quickly identify and prioritize the most critical vulnerabilities and threats. Impacted applications, version tracking, and remediation candidates provide insight into the impact and the actionable steps to take in order to keep the application secure. Vulnerabilities The Vulnerabilities page provides a complete view of the security of your services, focusing on the types and instances of vulnerabilities instead of on each library. It includes a real-time trend graph that shows the number of both fixed and open vulnerabilities.  Vulnerabilities are displayed by severity level, and the Severity Trend chart summarizes the trend of fixing open vulnerabilities. There’s also a Days Since First Detected chart showing the number of days the vulnerability is open versus the severity.  All of this detailed vulnerability information provides fast impact analysis and in-depth information on each vulnerability along with suggested fixes through remediation candidates.  Attacks The Attacks page is critical to providing information about attacks against your services. With real-time threat detection, you can get a high-level overview of security threats and the applications that they impact.  Attacks By Outcome represents the total number of attacks based on their states, and the Top Applications chart displays the top 10 applications based on attacks per application.  Attacks are also categorized for clear breakdown of the nature of the attacks. Attack types include SQL Injection, where an attacker is attempting to use unsanitized input to execute custom SQL statements to expose or manipulate private data, Remote Code Execution (RCE), where an attacker is attempting to gain access to a system and execute custom code, and others like:  DESERIAL LOG4J SSRF MALIP You can dig into specific attack details to view things like stack traces, source details, and the impacted business transaction: Note: If you have Configure permission for Cisco Secure Application, you can create or customize policies for vulnerabilities and attacks to specify which runtime behaviors to ignore, detect, or block. Cisco Secure Application Architecture and Installation How does this all work and how do you get started with Cisco Secure Application? According to the  Secure Application Architecture documentation: Install the supported APM Agent and then add the Cisco Secure Application license. The APM-managed application runs and the APM Agent retrieves the data to send to the Controller. The Cisco Secure Application service retrieves the application, tiers, and nodes data from the Controller. The APM Agent communicates with the Cisco Secure Application service to check if the security is enabled for the application. If the security is enabled, then the agent downloads the configuration along with the policies from the Cisco Secure Application service.  Based on the configured policies, the agent sends the security events to the Cisco Secure Application service. The service collects all the data, analyzes the application behavior, and then provides the analyzed data to the Cisco Secure Application dashboard.  Wrap Up Splunk AppDynamics with Cisco Secure Application provides real-time insight into the security health of your applications. Instead of waiting for a deploy to get visibility into vulnerabilities, Cisco Secure Application provides visibility at runtime and continuously detects and blocks attacks. This helps you quickly see and fix what matters to minimize business risk and easily prioritize what to work on.  If you’re ready to get started with Cisco Secure Application, you can first check out the requirements, install the agent that supports your environment, and then contact Splunk AppDynamics sales representative or email salesops@appdynamics.com to have a Cisco Secure Application license provisioned. More info can be found in the Application Security Monitoring Getting Started docs.    Resources Splunk Delivers Unified Security and Observability to Protect Applications Splunk AppDynamics Guided Tour Splunk AppDynamics Unveils New Security and Mobile Insights Splunk AppDynamics Application Security Monitoring ... View more

We’ve already explored a few topics around observability in a Kubernetes environment -- Common Failures in a Kubernetes Environment, how to Detect and Resolve Issues in a Kubernetes Environment, and the process of Integrating Kubernetes and Splunk Observability Cloud. But even though the process of setting up the OpenTelemetry Collector and exporting telemetry data to a backend observability platform like Splunk Observability Cloud can be completed in a few short steps, these initial setup steps don’t account for what happens when new workloads are spun up. Kubernetes is known for its dynamic nature, and observability needs to move right along with the ever-changing environment. In this post, we’ll see how monitoring Kubernetes workloads is currently done. Then we’ll check out how the new Kubernetes annotation-based discovery for the OpenTelemetry Collector makes it even easier. Monitoring Workloads: The Traditional Approach Monitoring dynamic workloads typically requires manual definitions of conditional configurations. Engineers define configuration “templates” based on environment conditions that adjust the monitoring configuration based on what’s automatically discovered in the environment. For example, to dynamically discover and monitor nginx pods that are deployed on the cluster, a receiver_creator definition is required:   This configuration is enabled when a pod is discovered via the Kubernetes API that exposes port 80 (HTTP port) and matches the nginx keyword. This approach works, but it requires a defined Collector configuration for each workload type. If new workload types are added to the environment, the configuration needs to be updated and redeployed, which can be time-consuming – especially if different teams are responsible for different aspects of the deployment. Additionally, it relies on the name of the pod, limiting flexibility to name workloads after what they actually do, making it harder to troubleshoot and diagnose problems. Monitoring Workloads: The New Approach The recently added Kubernetes annotation-based discovery for the OpenTelemetry Collector now allows for the dynamic configuration of monitoring workloads by adding annotations directly to their pods or namespaces. The K8s observer detects and reports information about objects appearing in the cluster to the Collector’s receiver_creator and automatically adjusts its configuration to monitor the annotated workloads. The annotations define the receiver that’s used to scrape telemetry data.  Here’s an example pod annotation that tells the OpenTelemetry receiver_creator to scrape metrics and enables discovery on the pod:    Finally, the discovery functionality must be enabled in the receiver_creator:   Once the deployment changes are applied and the Collector is restarted, you’re done! Adding a new nginx workload? No problem. Annotate your pods as shown above to dynamically configure monitoring for your new workload. It doesn’t matter what the pod is named and enables you to be more declarative about what is collected and how. Wrap up The amount of configuration maintenance required significantly decreases thanks to this new annotation-based discovery for Kubernetes and the OpenTelemetry Collector. Kubernetes observability becomes as dynamic as the environment it monitors, and configuration of dynamic workload discovery becomes a seamless experience. Already using the OpenTelemetry Collector in Kubernetes and want to try out this new feature? Want to export your automatically discovered Kubernetes workloads and their telemetry data to a unified observability platform? Check out Splunk Observability Cloud’s free 14-day trial. Resources Generate receiver configurations from provided Hints Kubernetes Monitoring and Alerting Made Easy with Splunk Observability Cloud and OpenTelemetry  Kubernetes Monitoring: The Ultimate Guide How to Monitor Kubernetes with Splunk Infrastructure Monitoring ... View more

Culture extends beyond work experience and coffee roast preferences on software engineering teams. Team culture also includes opinions on ticket creation/completion, merge request reviews, on-call rotations, and incident response. When it comes to incident response, a Don’t Repeat Incident (DRI) team or company culture can be hugely beneficial in maintaining a reliable code base, a positive user experience, and overall team happiness.  What is DRI? When an incident occurs, on-call team members are notified of triggered alerts. These individuals are often pulled into an incident response role to troubleshoot and resolve the incident. Traditionally, once the incident is resolved, the work of the on-call team members is over. Contrast this with a DRI approach: when the incident is successfully mitigated, the work is not over. Post-incident, work needs to be tasked out and prioritized to address any code changes or safeguards that need to be put in place to prevent the same incident from occurring again in the future – every incident results in the creation of one or more actionable tickets or issues. Sidenote: post-incident reviews (or postmortems) are also a great idea and serve as a retrospective around the incident to discuss why it occurred, how it could have been prevented, who was impacted, and how it will be prevented in the future.  When this whole process works, it helps create a proactive culture and reduces incident frequency. However, manual steps always leave room for error – a person forgets to create a ticket, a person forgets to follow up, and soon alerts for errors that have been seen before are firing yet again. Taking manual intervention out of the equation helps guarantee that tickets are created, added to the right repositories or boards, and prioritized to prevent incidents from repeating. How can technology help us with this? Cue webhooks. Webhooks Webhooks are a great way for systems to communicate when specific events occur. They can be used for a range of interactions – building out CI/CD pipelines, real-time data sharing like when sending confirmation emails after online order completion, initiating two-factor authentication at login, etc.   Because of their lightweight efficiency, many platforms support webhooks – Gmail, Slack, GitHub, Jira – the list is long. In this post, we’ll look at how Splunk Observability Cloud can help with the DRI practice described above to reduce incidents, increase code resiliency, and remove the need for manual intervention with webhooks.  Splunk Observability Cloud & Webhooks Your environment is unique, and Splunk Observability Cloud provides many integrations out-of-the-box (Slack, Jira, ServiceNow) that may suit your incident response needs. However, in situations where your environment needs aren’t met, custom webhook integrations have your back.  Say we keep track of our product’s code issues using GitHub issues. We can create a webhook integration in Splunk Observability Cloud to track active incidents by automatically opening an issue anytime an alert fires. Again, this helps ensure that human eyes land on the root cause and mitigation work is identified and recorded in an issue to make sure the incident does not repeat.   Let’s look at how we can go about setting up a GitHub webhook in Splunk Observability Cloud.  Configure a GitHub Webhook in Splunk Observability Cloud First, we’ll navigate to Data Management in Splunk Observability Cloud and search for the Webhook integration:    We can follow along with the guided setup and configure our GitHub connection:    Select Next to customize the auto-populated payload (but first, notice how much data you can send and act on in the remote system):    We’ll update the fields to match those required by the GitHub API to create a new issue and use the messageTitle and description variables to populate our issue:  After selecting Next we can review and save our webhook:  With our GitHub webhook saved, we next need to add it as a notification recipient on the desired detectors.  Note: to add a webhook as a detector recipient, you must have administrator access.  We can configure webhooks by editing the detector of choice:  We can add Alert recipients and select Webhook:   Then select our newly created GitHub Issue Webhook as the recipient and activate our updated alert notification:    Note: our GitHub webhook hits the GitHub REST API to create issues and these are scoped to the repository specified in the request URL – in our case, the Worms in Space repository. That means we would want to make sure the detector we attach this webhook to is related to that code repository specified in the URL. Thankfully, when we create detectors and alert rules, we can easily scope them to specific services.  That’s it! When an alert rule is triggered (and in this case when an alert is cleared or resolved), we’ll see a new issue automatically appear in our repo’s GitHub issues:  This is a simple example, but you can imagine how you could integrate this with a CI/CD system, for example, to monitor a critical metric, fires off a webhook that runs a script to determine if there’s been a recent release, and rolls back the release if the metric goes out of bounds. Exact examples vary a lot because there are so many systems out there, but the possibilities are really endless. Wrap up Now that our alerts create issues in our GitHub repository, we can prioritize and resolve their root cause and eliminate repeat incidents. Rather than being pulled away by reoccurring alerts, our focus can stay on delivering high-priority code, our applications can stay resilient, and our users can stay happy.  Want to accomplish something similar with Jira? Check out the Splunk Observability Cloud built-in Jira integration to easily connect Jira projects and create issues based on Splunk Observability Cloud alerts. Interested in building out a custom webhook? That works too! Once you’ve built a custom webhook, follow the same steps above so it can listen for and receive Splunk Observability Cloud alert notifications.  Don’t yet have Splunk Observability Cloud? Try it free for 14 days! Resources Send alert notifications to a webhook using Splunk Observability Cloud Send alert notifications to Jira using Splunk Observability Cloud Send alert notifications to services using Splunk Observability Cloud ... View more

Logs and Metrics Logs are generally unstructured text or structured events emitted by applications and written to files. Logs are essential to system visibility, especially when investigating unpredictable states of the system. Logs (ideally) give you data on exactly why something isn’t working. However, a logs-only approach becomes slow, expensive, and unsustainable as systems and the data they produce grow.   Metrics are data points that represent numbers, counters, or gauges that provide visibility into a system — error count, average CPU utilization percentage, throughput, etc. combined with a time stamp. Metrics are great for quick and reliable big-picture insight into system performance and can help teams quickly identify predictable system error states. But while out-of-the-norm metrics might indicate a problem within an application environment, it’s hard to pinpoint the root cause by looking at metrics alone.  Using a combination of logs and metrics provides many of the details needed to troubleshoot issues. So if we need both logs and metrics, what’s all this business about converting logs to metrics?  Why convert logs to metrics? In many systems, including Splunk, metric data points are stored in a format that provides faster search performance and more efficient data storage. This means log data that would be better represented by high-level, quantitative metrics rather than detailed text can be converted to metrics to help set alerting thresholds, reduce storage costs, and decrease the time it takes to troubleshoot issues. Rather than searching through logs of HTTP status codes and trying to parse patterns or identify outliers from the log text, we can convert status codes to metrics and easily view anomalies or even alert on them.  So why not just start off reporting these metrics? Maybe you’re already logging system data but are unable to alter the code base to instrument metric data – or perhaps you’re using a  commercial off-the-shelf (COTS) application that only emits logs. Maybe you need a low-barrier-to-entry way to gain deeper insight into your system with better search performance and lower data storage costs. Maybe certain teams need access to specific metric data but aren’t permitted to access the full details produced in logs. No matter the reason, pulling metric data from logs provides a higher-level system overview, a faster way to query data, and the ability to set alert thresholds on that metric data.  Convert logs to metrics with the OpenTelemetry Collector So how do we convert logs to metrics? Many observability platforms provide in-app ways to convert logs to metrics, so depending on which backend provider you use, the process can look different. In Splunk Enterprise, for example, ingest-time log-to-metrics conversion can be accomplished through Splunk Web or through configuration files.  For a quick and easy vendor-agnostic way to start reporting the metrics you need from the logs you already have, the OpenTelemetry Collector can be installed and configured to convert logs to useful metric counts. The Collector can be thought of as a pipeline to ingest logs from your application, process and extract metrics, and then send the metrics to an observability backend. Using the OpenTelemetry Collector approach allows us to:  Preprocess logs and offload the logs to metrics conversion from logging backends to the Collector  Send our metrics to locations outside of logging backends (like Splunk Observability Cloud) Reduce the need to store certain logs in logging platforms – instead, create the metrics and drop the logs or send them to low-cost archives Set our application up for future standardization by laying the groundwork for using OpenTelemetry Let’s take a look at how to do this.  Configuring the OpenTelemetry Collector Since we’re working with Splunk, we’ve installed the Splunk Distribution of the OpenTelemetry Collector. If you want to follow along and install the Collector yourself, you can work through the steps in the Splunk documentation to get started with the Splunk Distribution of the OpenTelemetry Collector.   With the Collector installed, we can configure the otel-collector-config.yaml file. First, we’ll update the receivers block to ingest our application’s log data:  This receiver sends our log data into the Collector just like it would to an HTTP Event Collector so no application code changes are required.  Next, we’ll need a connector. Connectors act as both an exporter and a receiver – they both consume and emit data. We’ll use a connector to consume our log data and then emit that data as a metric: The count connector is used to create a http_500_errors metric that counts any log events where the status code is equal to 500. The defined attributes will additionally create any dimensions we’d like to include on this count metric.  Note: we could also use the sum connector to convert our logs to sum attribute values. The connector you use depends on the type of metric you need and what you want the metric to represent. Jeremy Hicks wrote a super great, in-depth blog post on this called Introduction to the OpenTelemetry Sum Connector. Here’s a quick example of what using the sum connector would look like:  We’re going to move forward with the count connector example. We’ll want to export this new count metric to Splunk Observability Cloud, so we’ll also include an exporter for that:  If we wanted to send our logs to an HTTP Event Collector, we could also add another exporter here. In the future, we might even want to exclude exporting the logs we convert to metrics by including a filter processor here in our configuration.  Finally, we need to add these to the service pipelines: Here, we’ve configured our pipeline to receive logs from Splunk Enterprise, receive metrics from our count/http-error-500 connector, and export metrics to our otlp endpoint, aka Splunk Observability Cloud.  We can then save this configuration, start/restart the OpenTelemetry Collector, and search for our new metrics from the Metric Finder in Splunk Observability Cloud:  Wrap up Converting logs to metrics gives us a high-level view of our system and allows us to create dashboards, charts, and detectors to proactively detect and alert on anomalies. We can then use this information to dig into our logs for deep insight into the exact lines of code that are failing for super-fast issue resolution.  If you’re ready to convert your logs to metrics and visualize your metric data in an observability platform with unified, full-system visibility, start with a Splunk Observability Cloud 14-day free trial.  Resources Logs vs Metrics: Pros, Cons & When to Use Which From Webhook to Log and Log to Metric: OpenTelemetry Magic! Set up ingest-time log-to-metrics conversion in Splunk Web ... View more

Agent Saturation What and Whys In application performance monitoring, saturation is defined as the total load on a system or how much of a given resource is consumed at a time. If saturation is at 100%, your system is running at 100% capacity. This is generally a bad thing. Agent saturation is a similar concept. It represents the percentage of available system resources currently being monitored by an observability agent. 100% agent saturation means 100% of available resources are instrumented with observability agents, which is a great thing. When it comes to observability practices, agent saturation is how well a system is instrumented and can be represented by:  ( instrumented resources / total resources ) x 100 Because greater visibility into system health and performance means proactive detection of issues, improved user experience, more efficient troubleshooting, decreased downtime, and countless other pluses, 100% agent saturation is the ultimate goal.  So why doesn’t everyone get to 100% agent saturation for full system observability and magical unicorn application visibility status? It’s challenging! Setting up observability agents across distributed applications and environments takes time. In ephemeral, dynamic systems already integrated into existing solutions, it can just be too much of a lift.  But the good news is that if you’re already using Splunk (maybe for logging and security) there are quick and easy ways to improve your system observability. In this post, we’re going to look at how to leverage the Splunk Add-on for OpenTelemetry Collector to gain a quick win when it comes to improving agent saturation.  Improve Agent Saturation with the Splunk Add-on for OpenTelemetry Collector For Splunk Enterprise or Splunk Cloud Platform customers who ingest logs using universal forwarders, you can quickly improve agent saturation and deploy, update, and configure OpenTelemetry Collector agents in the same way you do any of your other technology add-ons (TAs). The Splunk Add-on for OpenTelemetry Collector leverages your existing Splunk Platform and Splunk Cloud deployment mechanisms (specifically the universal forwarder and the deployment server) to deploy the OpenTelemetry Collector and its capabilities for increased visibility into your system from Splunk Observability Cloud. The add-on is a version of the Splunk Distribution of the OpenTelemetry Collector that simplifies configuration, management, and data collection of metrics and traces. This means OpenTelemetry instrumentation will out-of-the-box exist anywhere the universal forwarder is present for logs and security use cases, making it easier to instrument systems quickly and gain visibility into telemetry data from within Splunk Observability Cloud. This comprehensive system coverage also comes with out-of-the-box Collector content and configuration with Splunk-specific metadata and optimizations (like batching, compression, and efficient exporting), all preconfigured. This means that you can get answers using observability data faster, saving you time and effort. Prerequisites for using the Splunk Add-on for OpenTelemetry Collector include:  Splunk Universal Forwarder (version 8.x or 9.x on Windows or Linux) Splunk Observability Cloud Splunk Enterprise or Splunk Cloud or deployment server as forwarders (Optional) Install the deployment server if you plan to use it to push the Collector to multiple hosts Getting started with the Splunk Add-on for OpenTelemetry Collector The Splunk Add-on for OpenTelemetry Collector is available on Splunkbase similar to other TAs, and you can deploy it alongside universal forwarders using existing Splunk tools like the deployment server.  We have a Linux EC2 instance we’re going to be instrumenting, but we first need to download the Splunk Add-On for OpenTelemetry Collector from Splunkbase:  We’ll unzip the package and then create a local folder and copy over the config credential files: In Splunk Observability Cloud, we’ll get the access token and the realm for our Splunk Observability Cloud organization:  Your organization's realm can be found under your user’s organizations:  Next, we set these values in our /local/access_token file: We then need to make sure the Splunk Add-On for OpenTelemetry folder (Splunk_TA_otel) is in the deployments app folder on the deployment server instance: We’ll then move over to the deployment server UI in Splunk Enterprise to create the Splunk_TA_otel server class and add the relevant hosts along with the Splunk_TA_otel app. Once the TA is installed make sure you check both Enable App and Restart Splunkd and select Save: That’s it! If we now navigate to Splunk Observability Cloud, we’ll see the telemetry data flowing in from our EC2 instance:  Wrap up Increasing agent saturation and improving observability for comprehensive system insight can be quick and easy. Not sure how you’re currently doing in terms of agent saturation? Check out our Measuring & Improving Observability-as-a-Service blog post to learn how to set KPIs on agent saturation. Ready to improve your agent saturation? Sign up for a Splunk Observability Cloud 14-day free trial, integrate the Splunk Add-on for OpenTelemetry Collector, and start on your journey to 100% agent saturation.  Resources Splunk Add-On for OpenTelemetry Collector Differences between the OpenTelemetry Collector and the Splunk Add-on for OpenTelemetry Collector Get started with Splunk Observability Cloud ... View more

Introduction In observability monitoring, static thresholds are used to monitor fixed, known values within application environments. If a signal goes above or below a static threshold or is within or outside a specified range, an alert will fire. Static thresholds are quick to configure and can provide helpful insight into system stats, but there are downsides to using them too frequently. There’s a time and a place for static thresholds, and in this post, we’ll look at when to use static thresholds, when not to use them, and alternatives to static thresholds.  When to use static thresholds Static thresholds work well for situations where there are “known knowns” – the predictable cases that you can anticipate – and when there’s a static range of “good” and “bad” values. Here’s an example of a CPU utilization detector with an alert rule that triggers when CPU is above 90% for a 5 minute duration:    As a side note, adding durations on such alert rules is important to avoid over-alerting on transient spikes. Without a set duration, every CPU spike over 90% would trigger an alert, as we can see when we try to configure the same condition without a duration:  The estimated alert count for this alert rule is 11 alerts in 1 hour – aka too much alert noise.  Boolean conditions, like monitoring synthetic test failures, are a great case for static thresholds. Alerting when a synthetic check fails indicates issues with site availability that should be addressed immediately. This detector alerts on a synthetic test when uptime drops below the 90% static threshold:    You can also use static thresholds to monitor Service Level Objectives/Service Level Agreements/Service Level Indicators (SLO/SLA/SLI). If you have an SLO of 99.9% uptime, you’ll want to be alerted anytime your availability approaches that threshold. Here’s an SLO detector that alerts if latency passes the 99.99999% threshold:    While configuring static thresholds on SLOs is possible, within Splunk Observability Cloud it’s recommended to manage SLO alerts using error budgets.  Static thresholds are most appropriate when working with fixed and critical metrics with clear failure conditions, e.g. HTTP error response codes spiking, response time increasing for a certain period of time, error rates above a certain percentage for a certain amount of time. If you’re just starting out on your observability journey, using static thresholds is also a great way to capture baseline metrics and gain insight into trends so you can fine-tune and adjust your detectors and alerts.  When not to use static thresholds As we saw in the CPU detector above, alerting on static thresholds can lead to a lot of alert noise if not used correctly. To create good detectors, we need to alert on actionable signals based on symptoms, not causes (we have a whole post on How to Create Good Detectors).  In dynamic system environments that include autoscaling and fluctuating traffic, static thresholds might not indicate actual system failures. Setting static thresholds on pod CPU in a Kubernetes environment, for example, could indicate increased load, but might not indicate a problem – pods could autoscale to handle the increase just fine.  Note: monitoring the CPU static threshold combined with pod queue length could provide the additional context needed to create an actionable detector.  When there are periods of traffic spikes – Black Friday for e-commerce sites, lunchtime for a food delivery site – setting static thresholds can lead to an increase in false alarms. Applications with such variable traffic fluctuations might not benefit from alerting on static thresholds alone.  Dynamic resource allocation and variable usage patterns can make using static thresholds in isolation tricky, but thankfully, alternative approaches can help. Alternatives to static thresholds For the situations mentioned above, along with those where we might not know all of our application’s failure states (unknown-unknowns), alternative detector thresholding or a hybrid approach can work best. Alternatives include but definitely aren’t limited to:  Ratio-based thresholds – instead of alerting on 500 MB of memory used, use a threshold of 80% of total memory for a specific duration of time Combining static thresholds with additional context – high CPU + pod queuing or error rate or latency  Sudden change detection – alerts on sudden changes to specified conditions like number of logins, response time, etc.  Historical anomaly detection to baseline environments and alert on deviations from trends – alerting on latency that deviates from historical trends Combining out-of-the-box approaches with custom thresholds can help you build a resilient monitoring solution to keep your applications running smoothly and your users happy.  Wrap up  For predictable, critical metrics static thresholds are a great alerting option. For situations when static thresholds aren’t appropriate, there are thankfully additional solutions that can help. To start, check out Splunk Observability Cloud’s out-of-the-box alert conditions here and explore what might work best for your unique application environment. Don’t yet have Splunk Observability Cloud? Try it free for 14 days!  ... View more

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth analysis on your incoming telemetry data. SignalFlow is the computational backbone that powers all charts and detectors in Splunk Observability Cloud, but the statistical computation engine also allows you to write custom programs in the SignalFlow programming language (modeled after and similar to Python). Writing SignalFlow programs definitely isn’t required for a complete observability practice, but if you want to perform advanced computations on your Splunk Observability Cloud data, SignalFlow is your friend. In this post, we’ll explore the whys and hows of SignalFlow so you can run these computations, aggregations, and transformations on your data and stream the results to detectors and charts for custom and in-depth observability analytics. Why use SignalFlow? Most Splunk Observability Cloud use cases don’t require complex computations. Out-of-the-box charts and detectors make building a complete observability solution easy. But sometimes there is a need for more detailed and advanced insight. SignalFlow can be used to: Define custom behavior or conditions for fine-tuned control over your monitoring so you can tailor your charts and detectors to your specific needs Aggregate metrics from different applications, cloud providers, or environments to unify data Troubleshoot by correlating metrics across many different sources – using SignalFlow during an incident helps provide deep, real-time investigation into root cause Detect trends over time or compare historical data to increase resiliency, reduce downtime, and capacity plan – i.e. correlate resources with user activity over time to optimize resource allocation Stream metric data to background analytics jobs – i.e. execute computations across a population over time Create reports or visualizations in third-party UIs so you can stream data out of Splunk Observability Cloud – i.e. in a service provider use case you can stream data to your own UIs and expose it to your customers Correlate business metrics with infrastructure and/or application metrics to understand how performance impacts customers – i.e. how does latency impact customer renewal rates There are many reasons why you might need to tap into SignalFlow; generally, if you need customized analytics, SignalFlow is the answer. So let’s see how it works! How to use SignalFlow You can define SignalFlow queries directly in the Splunk Observability Cloud UI or programmatically using the SignalFlow API. If you open up or create a chart in the UI, you’ll see the chart builder view:  If you select View SignalFlow, you can dive right into the SignalFlow that powers the chart and use it as a template for additional programs:  The same is true for detectors. If you open up a detector, you can select the kebab icon to Show SignalFlow:    SignalFlow programs outside of the Splunk Observability Cloud UI typically live within code configurations for detectors and dashboards (see our Observability as Code post). When you create a chart or detector using the API, you can specify a SignalFlow program as part of the request. Here’s an example of defining a detector using Terraform, where the program_text is the SignalFlow program:  You can also use the Signalflow API to run programs in the background and receive the results asynchronously in your client. Let’s take a look at some SignalFlow functions and methods we can use to build out charts and detectors.  SignalFlow Functions and Methods in Charts Most SignalFlow programs begin with a data() block. The data function is used to query data and is the main way to create stream objects, which are similar to a time-parameterized NumPy array or pandas DataFrame. Queries can run against both real-time incoming streams and historical data from the systems you monitor. In SignalFlow, you specify streams as queries that return data. Here’s a template for the data function:  We can expand on the data function in many ways. For example, here’s what it would look like to query for the CPU utilization metric and filter by host: We can also add/chain methods or functions to our data block. Here are examples of using the mean method to look at mean CPU utilization, mean CPU utilization by Kubernetes cluster, and mean CPU utilization over the last hour:  Operations like mean, variance, percentile, exclude, ewma, timeshift, rate of change, standard deviation, map(lambda), and others are available as methods on numerical streams. Here’s an example where our data stream, signal, is the CPU utilization with a filter of host, and we can add functions to timeshift by a week and two weeks, and then find the max value:  Comparing the max CPU utilization for two separate time series can’t actually be accomplished using the chart plot editor in the Splunk Observability Cloud UI, so this is an instance of where using SignalFlow is necessary.  To actually output these stream results to a chart, we need to call the publish() method:  We’ve now built out a chart using SignalFlow 🎉! We can also do this with detectors – read on. SignalFlow Functions and Methods in Detectors Detectors evaluate conditions involving one or more streams, and typically compare streams over periods of time – i.e. disk utilization is greater than 80% for 90% of the last 10 minutes. When building detectors using SignalFlow, we still start with our data streams, and then transform our data streams using boolean logic:  Note: when setting static thresholds in the UI, thresholds can only be greater than or less than. But as you can see with SignalFlow, we can specify greater than or equal to static thresholds.  We can use these when statements on their own or combined with and, or, not statements to publish our alert conditions and build out our detectors:  The detect streams in this example are similar to data streams. Detect streams turn our boolean statement – when our signal is greater than 90 for 1 minute – into an event stream. When this statement is evaluated as true, an event will fire and be published to an event stream. This is what triggers an alert.  Note: event streams are evaluated and published in real time as metrics are ingested, enabling you to find problems faster and speed up your MTTD. Every publish method call in a SignalFlow detect statement corresponds to a rule on the Alert Rules tab in the Splunk Observability Cloud UI. The label inside the publish block is displayed next to the number of active alerts in the Alert Rules tab:  You can create your detectors using the SignalFlow API, but if you want to use SignalFlow to build detectors directly in the Splunk Observability Cloud detector UI, you can append /#/detector/v2/new to your organization URL to do so:  Wrap up While working with SignalFlow is not required, it can help customize and advance your observability practice. A great place to start is editing the SignalFlow for existing charts and detectors in the Splunk Observability Cloud UI or using observability as code with SignalFlow programs. In no time, you’ll be building out SignalFlow program background jobs and streaming customized analytics to meet all your observability and business needs.  New to Splunk Observability Cloud? Try it free for 14 days!  Resources SignalFlow and analytics Analyze data using SignalFlow Up Close Monitoring with SignalFlow Intermediate to advanced SignalFlow ... View more

Observability protocols define the specifications or formats for collecting, encoding, transporting, and delivering telemetry data between telemetry data sources, intermediate sources like collectors, and telemetry backends like third-party observability platforms. The following table breaks down some of the most popular observability protocols, including OTLP, Prometheus, Jaeger, Zipkin, and others. Resources OpenTelemetry Protocol (OTLP) Prometheus StatsD OpenMetrics Jaeger Zipkin SysLog Fluent Forward  SNMP sFlow   ... View more

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to get the end-to-end insight we need to keep our applications healthy and our customers happy. In this post, we’ll explore how to integrate Amazon Elastic Kubernetes Service (EKS) with Splunk Observability Cloud so we can observe EKS alongside the rest of our application telemetry data.  Observability Metrics in AWS  The Amazon EKS management console and Amazon CloudWatch provide insight into EKS observability metrics. From the AWS management console, you can see things like cluster health and details about EKS resources deployed to your cluster. Node details are available within a selected cluster’s Resources tab:  You can dig into pod status, capacity, and pod details: Get an overview of node conditions like MemoryPressure, DiskPressure, etc.: You can even get insight into node events like node status:  For more detailed cluster health visibility, you can enable CloudWatch Observability from the EKS console. With Container Insights enabled, you get even deeper insight into cluster state and resource utilization:  Along with cluster, namespace, node, service, workload, pod, and container performance monitoring:  But if your infrastructure isn’t all in AWS and is spread across multiple platforms, navigating between observability tools, especially during a high-pressure incident, is not great. Instead, having one unified observability platform where you can view all these metrics and more reduces toil and time to incident resolution. End-to-end visibility unified in a central platform makes for a more resilient and efficient observability practice. So let’s look at how to integrate one such observability platform, Splunk Observability Cloud, with Amazon EKS.  Integrate AWS and Splunk Observability Cloud Splunk Observability Cloud provides a unified platform for troubleshooting and monitoring all application systems no matter where they live. Not only can you collect and store Amazon Cloudwatch Metrics data, but if pieces of your applications and infrastructure live outside of AWS, you can view that data right alongside your AWS data for a complete observability picture. You may have already integrated AWS with Splunk Observability Cloud through the Data Management section in Splunk Observability Cloud:  The integration wizard easily takes you through the process of preparing your AWS account:  And getting your AWS data flowing into Splunk Observability Cloud:  But for EKS, data is collected using the Splunk Distribution of the OpenTelemetry Collector, and even with AWS integrated with Splunk Observability Cloud, you’ll notice from the Available Integrations page that we still need to deploy the OpenTelemetry Collector to get our EKS data in:  So let’s install the Splunk Distribution of the OpenTelemetry Collector for Kubernetes.  Install the Splunk Distribution of the OpenTelemetry Collector We’ve gone through how to integrate Kubernetes and Splunk Observability Cloud before, and integrating Amazon EKS isn’t much different. When we follow along with the integration wizard, we just need to specify Amazon Web Services as the provider and Amazon EKS (or Amazon EKS / Fargate profiles) as the distribution:  We can connect to our EKS cluster and then follow along with the rest of the installation instructions. I’m using the AWS CLI in my terminal, but with Helm installed, you could also use AWS CloudShell. I first configured kubectl for my EKS cluster by updating my kubeconfig file:  And verified the connection:  I next ran the commands in the Splunk Observability Cloud installation instructions with splunk-otel-collector --version pinned to 0.111.0:  Once those steps were complete, I could then view my EKS telemetry data from within Splunk Observability Cloud:  In a previous post, we explored what it looks like to navigate Kubernetes data using Splunk Observability Cloud navigators to detect and resolve issues in a Kubernetes environment. Now that our EKS cluster is sending data to Splunk Observability Cloud, we can use all the same products and features within Splunk Observability Cloud to monitor our Amazon EKS environment.  From Infrastructure Monitoring we can view our Amazon EKS navigators:  We can get insight into all of our Kubernetes clusters:  Dive into the health of a specific cluster: And observe critical performance data around nodes, containers, daemonsets, deployments, namespaces, pods, replicasets, and workloads:  From these critical usage metrics, we can create detectors and alerts from within our navigators and they can live right alongside the detectors and alerts for the rest of our applications and infrastructure: With Amazon EKS now successfully integrated, we can use Splunk Observability Cloud to proactively monitor, detect, and alert on anomalies in our EKS environment right alongside the rest of our application and infrastructure telemetry data.  Wrap up Integrating with a third-party observability platform like Splunk Observability Cloud provides a unified observability solution for your applications and infrastructure. This helps with quick and easy incident detection and resolution without having to navigate between a bunch of different observability solutions.  Want to try integrating Amazon EKS with Splunk Observability Cloud? Try Splunk Observability Cloud free for 14 days!  Resources Get started with the Collector for Kubernetes Available Amazon Web Services integrations ... View more

At their best, detectors and the alerts they trigger notify teams when applications aren’t performing as expected and help quickly resolve errors to keep things up and running. At their worst, they cause an overwhelming amount of noise that fatigues our engineering teams, distracts from real application issues, and decreases the reliability of our products. In this post, we’ll look at ways we can skew toward the former to keep our detectors helpful, our engineers unpaged by unnecessary alerts, our applications performant, and our users happy.  What are detectors? Detectors set conditions that determine when to alert or notify engineering teams of application issues. These detectors monitor signals and trigger alerts when the signals meet specified rules. These rules are defined either by engineering teams or automatically by third-party observability platforms, like Splunk Observability Cloud’s AutoDetect alerts and detectors.  It can be tempting to create detectors around every possible static threshold that might signal that our system is in an undesirable state – CPU above a certain percentage, disk space low, throughput high, etc. However, creating too many detectors and setting off too many alerts doesn’t always lead to actionable solutions and causes alert fatigue. Creating the right detectors can help minimize false positives or negatives, reduce alert noise, and improve troubleshooting while reducing downtime and fatigue.   How to create good detectors It’s important to thoughtfully and intentionally create detectors to keep them as meaningful and helpful as possible. Detectors and the alerts they trigger should be:  Signals of an actual emergency or a signal that there’s about to be an emergency Actionable and based on symptoms, not causes Well-documented Built using Observability as Code Iterated upon  Let’s dig into each of these best practices.  Detectors should signal an actual emergency or impending emergency The Google SRE book defines alerting as, “Something is broken, and somebody needs to fix it right now! Or, something might break soon, so somebody should look soon.” Preferably, our system is self-healing and can autoscale, load balance, and retry to minimize human intervention. If these things can’t happen, and our users aren’t able to access our site, are experiencing slow load times, or are losing data, then our teams should be paged because there’s a real emergency that requires human intervention.  If the alert being fired doesn’t meet these criteria, it should perhaps be a chart living in a dashboard or a factor contributing to the service’s health score instead of an alert that could potentially wake engineers up in the middle of the night. For example, high throughput could lead to slower response times and a degraded user experience, but it might not (e.g. when CPU spikes precede auto-scaling events). Instead of creating a detector around high throughput itself, alerting on throughput thresholds and metrics surrounding latency or error rates would indicate an official or looming emergency.   At the end of the day, your application exists to serve a purpose. If that purpose isn’t being served or is in danger of not being served, alerts need to fire. If your infrastructure is running extremely hot, but the application is working, you can probably wait on alerting until business hours to look at additional capacity. But if users are being impacted, that’s a real emergency.  Detectors should be actionable Human intervention is expensive. It interrupts work on deliverables, distracts from other potential incidents, and can unnecessarily stress out and exhaust the humans involved – either by literally waking them from sleep or by fatigue from too many alerts. Therefore, when detectors and the alerts they trigger are created, they should define actionable states.  If a detector sets a static threshold on something like CPU and creates an alerting rule that triggers an alert when CPU spikes above 80%, the human responding to that page will go into an observability platform, see the CPU spike, and may not have much more information right off the bat. Are error rates spiking? Are response times slow? Are customers impacted? Is CPU spiking because of garbage collection or backup processing? Is any action necessary? Static thresholds like these don’t provide much context around end-user impact or troubleshooting. Instead, creating detectors and alerts around the symptoms or the things that impact user experience (think Service Level Objectives), means that there’s something wrong, there’s a real emergency, and a human needs to step in and take action.  Some examples of actionable symptoms include:  HTTP response codes like 500s or 404s Slow response time Users aren’t seeing expected content Private content is exposed to unauthorized users  Here’s a good example of a Splunk Observability Cloud detector that alerts when the error rate for the API Service is above 5% over a 5-minute time period:  Alerts would be triggered where the red triangles are shown.  Another example of a seemingly good detector is one that alerts when disk utilization percentage for a specific host goes above 80% for a specified amount of time:  However, this specific host initiates a data dump to S3 every 48 hours at 4 am EST, and this data dump causes a period of increased disk utilization. We can see in the alert settings that this alert is predicted to trigger in 48 hours when our S3 dump will occur, and we definitely don’t want to page the engineering team that manages this host at 4 am. We could adjust the singal resolution to a longer time period if needed, or we could create a muting rule that suppresses notifications for this specific condition:  The downside to this is that we could potentially miss a real incident, so we might also want to create a second detector around disk utilization percentage with a slightly higher threshold (that wouldn’t alert with our data dump spike) to monitor utilization during this time period.  Detectors should be well-documented Responding to an alert triggered by a detector should not require institutional knowledge and pulling multiple humans in to resolve the incident. There are times when this will happen, but detectors should be well-documented with runbooks for quick and easy guidance on troubleshooting steps.   Notifying the right people and setting the right escalation policies should also be considered in the configuration of detectors and alerts. For example, it doesn’t make sense to page the entire engineering team for every incident; it does make sense to page the code owners for the service experiencing issues. It doesn’t make sense to page leadership teams if a handful of customers are experiencing latency; it does make sense to page leadership teams if an entire product is offline.  Detectors should be built using Observability as Code Speaking of documentation… creating detectors and alerts as code provides a source-controlled documentation trail of why, when, and by whom the detector was created. This means that any new members joining the team will have background knowledge on the detectors and alerts that they will be on-call for. It also creates an audit trail when fine-tuning and adjusting thresholds, rules, notification channels, etc.   Using Observability as Code also makes it easier to standardize detectors and alerts between services, environments, and teams. This standardization means that all members of all teams have at least a baseline knowledge of the detectors and alerts and can potentially have greater context and ability to jump in and help during an incident.  Here’s an example of creating a Splunk Observability Cloud detector using Terraform:  Check out our previous post to learn more about deploying observability configuration via Terraform.  Detectors should be iterated upon  Detectors and alerts should work for our teams, not against them, and they should constantly evolve with our services to improve the reliability of our systems. To keep detectors and alerts actionable and relevant, it’s important to update, fine-tune, or even delete them when they’re no longer effective or helpful. If alerts are frequently firing but require no action, this is a sign to adjust thresholds or re-evaluate the need for the detector. If the actions taken by the incident response team could be easily automated, this is a sign to adjust or remove the detector. If anyone on your team uses the word “ignore” when describing a detector or alert, this is a sign to adjust or remove the detector. If team members are unable to troubleshoot an incident effectively based on the information they receive from a detector or alert, this is a sign to iterate on the detector.  Alerting detectors should never be the norm. If a detector is alerting, action should be required. Having a board full of “business as usual” alerting detectors quickly leads to false positives, false negatives, and ultimately hurts customer experience. Wrap up Detectors and alerts are critical to keeping our systems up and running, but too many detectors can work against that goal. Good detectors:  Signal real emergencies  Indicate an actual issue with user experience and provide our on-call teams with actionable insight Are well-documented Are built using source control Are continuously fine-tuned  These things help keep our detectors and alerts useful, our engineering teams focused, and our users happy.  Interested in creating some good detectors in Splunk Observability Cloud? Try out a free 14-day trial! Resources Google SRE book: Monitoring Distributed Systems How to Create Great Alerts Introduction to alerts and detectors in Splunk Observability Cloud Introduction to the Splunk Terraform Provider: Create a Detector in Splunk Observability Cloud ... View more

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to collect and monitor for database performance, health, and reliability. In this post, we’ll look at how to monitor MariaDB (and MySQL) so we can keep the data stores that back our business applications performant, healthy, and resilient.  How to get the metrics The first step to get metrics from MariaDB or MySQL (for the rest of this post, I’ll just mention MariaDB, but the same approach and receiver works for both) to Splunk Observability Cloud is to instrument the backend service(s) that are connected to your MariaDB(s). If you’re working with the Splunk Distribution of the OpenTelemetry Collector, you can follow the guided install docs. Here I’m using Docker Compose to set up my application, MariaDB service, and the Splunk Distribution of the OpenTelemetry Collector:   Next, we’ll need to configure the OpenTelemetry Collector with a receiver to collect telemetry data from our MariaDB and an exporter to export data to our backend observability platform. If you already have an OpenTelemetry Collector configuration file, you can add the following configurations to that file. Since I set up the Collector using Docker Compose, I needed to create an empty otel-collector-config.yaml file. The mysql receiver is the MariaDB-compatible Collector receiver, so we’ll add it under the receivers block along with our Splunk Observability Cloud exporter under the exporters block. Here’s what our complete configuration looks like:  As always, don’t forget to add the receivers and exporters to the service pipelines.  Done! We can now build, start, or restart our service and see our database metrics flow into our backend observability platform.  Visualizing the Data in Splunk Observability Cloud With our installation and configuration of the OpenTelemetry Collector complete, we can now visualize our data in our backend observability platform, Splunk Observability Cloud.  From within Application Performance Monitoring (APM), we can view all of our application services and get a comprehensive performance overview – including performance details around our MariaDB:  We can explore our Service Map to visualize our MariaDB instance and how it fits in with the other services in our application: And we can select our MariaDB instance to get deeper insight into performance metrics, requests, and errors:  If we scroll down in the Breakdown dropdown results on the right side of the screen, we can even get quick visibility into Database Query Performance, showing us how specific queries perform and seeing which queries are being executed:  Clicking into Database Query Performance takes us to a view that can be sorted by total query response time, queries in the 90th percentile of latency, or total requests so we can quickly isolate queries that might be impacting our services and our users: We can select specific queries from our Top Queries for more query detail, like the full query, requests & errors, latency, and query tags:  We can dig into specific traces related to high-latency database requests, letting us see how specific users were affected by database performance:  And also see right down into the span performance: And the db.statement for the trace: We can proactively use this information to further optimize our queries and improve user experience. You can also see that the database activity shows up in the overall trace waterfall, letting you get a full picture of how all components of the stack were involved in this transaction. But how can this help us in an incident? This is all helpful information and can guide us on our journeys to improve query performance and efficiency. But when our database connections fail, when our query errors spike, that’s when this data becomes critical to keeping our applications up and running.  When everything is running smoothly, our database over in APM might look something like this:  But when things start to fail, our Service Map highlights issues in red:  And we can dive into traces related to specific error spikes:  The stacktrace helps us get to the root cause of errors. In this case, we had an improperly specified connection string, and we can even see the exact line where an exception was thrown:  With quick, at-a-glance insight into service and database issues, we easily jumped into the code, restored our database connection issue, and got our service back up and running so our users could carry on with enjoying our application.  Wrap Up Monitoring the datastores that back our applications is critical for improved performance, resiliency, and user experience. We can easily configure the OpenTelemtry Collector to receive MariaDB telemetry data and export this data to a backend observability platform for visibility and proactive detection of anomalies that could impact end users. Want to try out exporting your MariaDB data to a backend observability platform? Try Splunk Observability Cloud free for 14 days!  Resources Database Monitoring: Basics & Introduction OpenTelemetry Collector Configuring Receivers Monitor Database Query Performance ... View more

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about integrating cloud-managed Kubernetes platforms? In this post, we’ll dig into how to monitor Google Kubernetes Engine (GKE) by integrating with Splunk Observability Cloud.  Wait, but why? GKE dashboards provide observability metrics around infrastructure and application health for clusters and workloads from within the Google Cloud Platform (GCP) itself. So why should we integrate GKE with an observability backend? Typically, not every piece of application infrastructure lives in the same cloud platform. In the middle of an incident when seconds matter, no one wants to navigate between platforms and remember where each discrete observability tool lives. Instead, having everything in one unified observability platform reduces toil and time to incident resolution. With end-to-end observability all in one place, root-cause analysis becomes a lot easier thanks to the ability to correlate issues impacting multiple parts of the stack. Third-party observability platforms provide advanced, flexible, and configurable dashboards, charts, and detectors along with support for application, service, and incident management integrations. Plus, depending on which platform you choose, you can avoid vendor lock-in by using an OpenTelemetry-native observability platform.  Observability Metrics in GKE We can monitor our applications running in Google Kubernetes Engine from GCP itself by configuring the Kubernetes Engine Monitoring setting on our cluster to System and workload wogging and monitoring. We can click into our deployed workload and see the CPU, memory, and disk utilization: And dig into the observability details around the health of our application – like container errors and restarts:  We can even go into Google Observability Monitoring and further explore GKE cluster performance: And build custom dashboards and charts:  These things are all great and useful, but again, to unify our observability tools and improve the time to incident resolution, how can we get all of this information into a single backend observability platform? Integrate GKE and Splunk Observability Cloud There are a couple of ways you can integrate GKE into Splunk Observability Cloud. We’ll first follow along with the Connect to Google Cloud Platform documentation.  Integrate Google Cloud Platform From within the GCP UI, we’ll go into the project we want to monitor and add a new Splunk Service Account under IAM & admin: Once the new service account is created, we can edit it to create a new service account key:  And then download the key as JSON:  Note: GCP Cloud Resource Manager API must be activated so Splunk Infrastructure Monitoring can use it to validate permissions on the service account key. Also, if you want to monitor multiple GCP projects, the above steps need to be repeated for each one. Next, we’ll navigate to Splunk Observability Cloud and add our Google Cloud Platform integration:  We can then import our newly created service account key: A benefit of integrating GCP in this way is that we can sync all supported GCP services automatically. However, for our purposes, we’ll optionally refine the GCP synced services to only include GKE:  Once saved, we’ll see Google Cloud Platform under our actively deployed integrations:  Depending on the polling interval, it might take a few minutes for our GCP metric data to populate within Splunk Observability Cloud. Integrate Google Kubernetes Engine  If you’re only interested in integrating GKE without additional GCP service integration (or if your security team won’t give you a service account), you can integrate with Splunk Observability Cloud via Helm chart. We can search for Google Kubernetes Engine in the list of available integrations and follow the guided Kubernetes Monitoring instructions: We’ll specify Google Cloud Platform as the provider and Google GKE as the distribution:  And then we’ll follow along with the installation instructions: In the GKE UI, activate a cloud shell, and then enter the above commands within the instance:  You’ll notice our helm-install splunk-otel-collector command failed because of a missing instrumentation.endpoint value. This issue results from the fact that the version of helm pre-installed in the Google Cloud Shell is out of date. To resolve the error, we need to upgrade the helm version to one supported for the current Kubernetes version and rerun the commands in the Splunk Observability Cloud installation instructions:  After this completes successfully, we can go back over to Splunk Observability Cloud and see our integrated telemetry data: From here, we can explore data like CPU and memory usage/utilization, resource capacity, container restarts, node state, and others right alongside the rest of our application and infrastructure data within Splunk Observability Cloud, even if other parts of the app run on-premises or on another public cloud provider.   Observability Metrics in Splunk Observability Cloud Now that our GKE integration is complete, we can use all of the available Splunk Observability Cloud products and features to monitor our GKE environment. From Infrastructure Monitoring we can view our Google Cloud Platform navigators:  And dive into the health of our clusters:  Dig into a specific cluster:  Monitor pod health and performance:  And view critical usage metrics:  We can create detectors and alerting rules from within our Navigators and manage them alongside the detectors for the rest of our applications and infrastructure: In a previous post, we looked in detail at how to use the Kubernetes Navigators to Detect and Resolve Issues in a Kubernetes Environment. With GKE now integrated with Splunk Observability Cloud, we can use these same tools to proactively monitor, detect, and alert on anomalies in our GKE environment.   Wrap up  We can monitor our GKE environment from within GCP itself, but integrating with a backend observability platform like Splunk Observability Cloud unifies our monitoring solution. With one unified observability platform, we can more easily detect incidents and resolve them faster without having to navigate between different observability tooling. Want to try integrating GKE with Splunk Observability Cloud for yourself? Try Splunk Observability Cloud free for 14 days! Resources Connect to Google Cloud Platform: Guided setup and other options Install the Collector for Kubernetes using Helm Splunk OpenTelemetry Collector for Kubernetes ... View more

We can’t guarantee the health of our services or a great user experience without data from our applications. Is CPU usage high? Are there an increased number of requests? Do we have too many Kubernetes nodes stuck in a NotReady state? These metrics and many others impact our services and our customers, but if we can’t see them, we can’t fix them. So, we build out an observability practice, instrument our services, collect all the metrics, export metrics to an observability backend, and quickly realize that our systems produce an overwhelming amount of data. We store metrics to understand trends, but data storage costs money so collecting and storing everything quickly becomes a budgetary problem. In the noise of all that data, it’s also difficult to identify which metrics are relevant in determining what is actually negatively impacting our services and users.  In a world where setting up a functional observability practice is as easy as installing the OpenTelemetry Collector configured with auto-instrumentation, there are different ways to manage metric pipelines so that metric collection is sustainable and cost-effective and provides real value to the reliability of our applications. In this post, we’ll look at how OpenTelemetry processors specifically can help manage data from within metric pipelines to avoid exporting and storing unhelpful data so you can focus on service reliability, faster troubleshooting, and lower observability costs. Managing Metric Data Volume Best Practices The following best practices help eliminate metric noise, reduce metric collection volume, and ensure helpful metrics are available and ready to support troubleshooting efforts.  Use OpenTelemetry Semantic conventions for metric names and attributes Collect metrics intentionally  Monitor the pipeline itself Optimize the pipeline and exporting processes Let’s take a look at each of these. OpenTelemetry Semantic Conventions Using OpenTelemetry metrics semantic conventions when naming metrics or metric attributes helps with data analysis and troubleshooting. Defining clear metric names and attributes also helps identify redundancies or commonalities between metrics. Without the use of semantic conventions, different engineering teams might use different names for the same metric, leading to metric redundancy and increased data volume. For example, metrics around total HTTP requests could be named: http_requests_total, total_http_requests, http_request_count, etc. With semantic conventions in place, these individual metrics can be consolidated into one single, shared metric like http.server.requests, which captures aggregated total requests and attributes like request method and endpoint. When metrics follow naming conventions, aggregations, filters, and transformations can more easily be applied to reduce the volume of metric data, reduce the cost of backend platform storage, and improve the effectiveness of observability practices.  Collect and Store Metrics Intentionally With semantically named metrics, it’s easier to identify those that provide value and rename or remove the ones that don’t, but how do you determine which metrics are and are not helpful? Here are some questions to consider: Could the metric be used for an actionable, high-priority alert?  Would the data reported by the metric create a meaningful dashboard? Is the individual metric meaningful? Or would an aggregation be more impactful?  It’s also important to note that not all metrics need or should be exported to backend observability platforms – not all data is relevant for troubleshooting or development purposes and doesn’t need to be readily available within observability backends. Cold data that won’t actively be used, like metrics necessary for compliance or audit purposes, can be exported to backend storage like Amazon S3 (perhaps even Glacier). This can lower storage costs and keep observability backends clear of metrics that aren’t immediately helpful in monitoring the resiliency and performance of applications.   Monitor the pipeline Monitoring the pipeline itself (e.g. Collector performance and resource limits) can help identify delays and/or constraints in processing or exporting. This ensures data integrity and quick insight into any issues with metric collection. It also provides insight into the performance and effectiveness of metric collection so you can iterate on which metrics you’re collecting and how you’re collecting them. Optimize the pipeline and exporting process Optimizing the pipeline collection and exporting processes ensures efficient data flow from collection to the backend platform so you can prevent bottlenecks and delays and successfully use the metrics you collect for performance monitoring and troubleshooting. OpenTelemetry Processors So how do you put these best practices into… practice? The OpenTelemetry Collector provides several processors that can be configured to transform data before it’s sent to observability platform backends. We can think of these processors more as pre-processors, taking many points of data and interpreting or condensing them into more meaningful information. Processors offer more control over metric collection so data can be reported in useful ways that reduce metric noise and storage costs.  Filter Processor Metric data can be included or excluded through configuration of the filter processor in the OpenTelemetry Collector configuration file. Any low-priority or unhelpful metrics, like those with invalid types or specified values, can be filtered out. Here’s an example that shows how to drop an HTTP healthcheck metric:  This metric doesn’t provide meaningful or actionable data around application performance or reliability, so to reduce metric volume and storage costs, we can drop it before exporting it to our backend observability platform. Attribute and Metric Transform Processors The OpenTelemetry attribute and metrics transform processors can be configured to modify and/or consolidate metrics. Their functionalities overlap a bit – you can add attributes or update attribute values using either processor. The metric transform processor provides more room for data manipulation, and the docs recommend that if you’re already using the metrics transform processor functionality, there’s no need to switch over to the attribute processor.  The metric transform processor can be used for renaming metrics so you can modify metric names or attributes while sticking to semantic conventions to reduce the number of discrete but related metrics, which are often billed separately in observability backends. For example, when using multiple cloud providers like Amazon Web Services (AWS) or Google Cloud Platform (GCP), each provider reports CPU utilization data under different metric names. Instead of reporting these metrics separately, they can be combined into a single metric name following semantic conventions to reduce cardinality and improve metric management. Here’s an example of updating AWS and GCP CPU utilization metrics to report under a single cloud.vm.cpu.utilization metric with attributes indicating the cloud provider and cloud service:  Group by Attributes Processor To organize metric data and more easily apply aggregations and transformations to specific groups of metrics, use the group by attributes processor. For example, if you’re collecting data from multiple services each running on multiple instances, the group by attribute processor can be configured to group metrics by service name as follows: Aggregations can be applied to sum or average metrics for each instance of each service. If service_a is running on instance_1, instance_2, and instance_3, we can use the group by attributes processor to combine these individual instance metrics into one single aggregated service_a metric. This reduces cardinality and data volume, while also making the metric data easier to troubleshoot.  Batch Processor While the batch processor doesn’t manipulate the raw metric data itself, it does contribute to an effective metrics pipeline by improving export performance. Effective exporting of our data means it gets to where it needs to go and is readily available for use when we need it. Batching metrics to compress data reduces the number of outgoing connections in order to improve exporting performance. It can easily be configured within the Collector configuration file by specifying batch under the processors block:  Or additional configuration options like batch size and timeout can be specific for more fine-grained control:  Memory Limiter Processor Like the batch processor, the memory limiter processor is related to the overall functionality of the metric pipeline. Using this processor ensures metric collection functions properly and data is collected, processed, and exported successfully. The memory limiter processor performs periodic checks of memory usage to prevent the Collector from running out of memory. If the Collector hits memory limits, it will start refusing data and lead to metric data loss. Here’s how to configure the memory limiter with a couple of the available options within the Collector configuration file: Wrap Up Observability isn’t about collecting all of the data. Collecting all the data is counterproductive to maintaining reliable systems that successfully support our customers. Instead, observability is about surfacing and analyzing actionable data. Managing metrics data volume at the point of collection with OpenTelemetry processors can help reduce the noise making it easier to detect anomalies and resolve issues faster. OpenTelemetry, a native part of Splunk Observability Cloud, provides built-in metric pipeline management and one unified observability backend platform for profiles, metrics, traces, and logs – no third-party pipeline management tools required. With Splunk Observability Cloud, you can also manage some metrics after ingestion with Metrics Pipeline Management (MPM). Interested in reducing metric volume, storage costs, and troubleshooting efficiency? Start a Splunk Observability Cloud 14-day free trial and adopt OpenTelemetry to tame your metric pipeline and to experience the benefits of one centrally-located backend observability platform (aka Splunk Observability Cloud). Using the power of Splunk Observability, watch your metric data flow in and your data storage costs go down (all while optimizing troubleshooting and reducing time to resolve incidents thanks to helpful and well-managed metric data).  Resources OpenTelemetry Metrics Transforming telemetry OpenTelemetry Collector Configuration Data Storage Costs Keeping You Up at Night? Meet Archived Metrics ... View more

Behind every business-critical application, you’ll find databases. These behind-the-scenes stores power everything from login and checkout to content lookups and “likes,” so issues with slow queries, too many full table scans (or too few index scans), incorrectly configured indices, or resource exhaustion directly impact application reliability and user experience. Thankfully, we can capture key database metrics to expose such issues and ensure optimal performance, efficient troubleshooting, and the overall reliability of our applications.  In this post, we’ll explore monitoring the open-source relational database PostgreSQL. Postgres is widely used in enterprise applications for its scalability, extensibility, and support. It also collects and reports a huge amount of information about internal server activity with its statistics collector. We’ll harness these stats using the OpenTelemetry Collector and first focus on the database and infrastructure itself in Splunk Observability Cloud. Then we’ll see how everything connects to our application performance data. Which metrics matter and why Monitoring database metrics is critical to proactively identifying issues, performance optimizations, and database reliability, but with so many stats coming from the statistics collector, it can be difficult to determine what to focus on. How do we isolate what’s critical to monitor effectively? It can help to focus on operation-critical key metrics like those related to: Query performance (query throughput/latency, locks, query errors, index hit rate) Resource utilization (connections, CPU, memory, disk space, table/index size, disk I/O, cache hits) Database health (replication lag, deadlocks, rollbacks, autovacuum performance) Query Performance Slow, resource-intensive queries or queries with high throughput can decrease the response time of our applications and degrade user experience. To prevent things like slow page load time, we want to focus on metrics related to query time –  total response time, index scans per second, and database latency. These metrics will indicate if our database has the right or wrong indexes, absent indexes, if our tables are fragmented, or have too many locks, etc.    Resource Utilization Exceeding resource thresholds can halt application operations altogether. If total active connections are too high resources might be exhausted, and users might not be able to interact with our application at all. Monitoring resource usage like CPU, memory, and table/index size can keep our databases up and running, while also allowing for accurate capacity planning and optimal user experience.  Database health Things like a high rollback to commit rate can indicate user experience issues, for example, users might be unable to complete product checkout on an e-commerce site. An increase in the number of dead rows can lead to degraded query performance or resource exhaustion with similar effects. Proactively monitoring these metrics helps easily identify inefficiencies, eliminate bottlenecks, reduce database bloat, and ultimately improve user experience.   How to get the metrics So how do we get these metrics from PostgreSQL to the OpenTelemetry Collector? The first step is installing the OpenTelemetry Collector. If you’re working with the Splunk Distribution of the OpenTelemetry Collector, you can follow the guided install docs. I’m using Docker Compose to set up my application, Postgres service, and OpenTelemetry Collector, so here’s how I added the Splunk Distribution of the OTel Collector:  If you already have your OpenTelemetry Collector configuration file ready to edit, you can proceed to add a PostgreSQL receiver to the receivers block so you can start collecting telemetry data from Postgres. Because I set up the Collector with Docker Compose, I manually created my Collector configuration file (otel-collector-config.yaml). Here’s the PostgreSQL receiver I added to my Collector config:  Note: generally, your database and microservices would be behind network and API security layers so your databases and services would talk to each other unencrypted, which is why I have tls set to insecure: true. If your database requires an authenticated connection, you’ll need to supply a certificate similar to what’s shown in the documentation’s sample configuration.  I’m also exporting data for my application to my Splunk Observability Cloud backend, so I’ve added an exporter for that and added both my new receiver and new exporter to my metrics pipeline:  If you’re not using the Splunk Distribution of the OpenTelemetry Collector or not exporting data to Splunk Observability Cloud, configuring the PostgreSQL receiver block will still follow the example shown, but you’ll need to configure a different exporter and add it to the metrics pipeline.  That’s it! Now either build, start, or restart your service (I did a docker compose up --build) and watch your database metrics flow into your backend observability platform of choice.  Note: If you’re working with a complex service architecture and the Splunk Distribution of the OpenTelemetry Collector, you might want to consider using automatic discovery. This allows the Collector to automatically detect and instrument services and their data sources. Depending on your environment and Collector installation method, you can follow the appropriate docs (Linux, Windows, Kubernetes) to deploy the Collector with automatic discovery. How to see the data in Splunk Observability Cloud Now that we’re collecting Postgres data, let’s jump over to Splunk Observability Cloud Infrastructure to visualize our telemetry data. We can select the Datastores section and open up either our PostgreSQL databases for database-level metrics or PostgreSQL hosts for metrics related to the infrastructure hosting your PostgreSQL database(s): Going into the PostgreSQL databases navigator, we can see the metrics related to all of our databases: Here we see those key metrics that can hint at performance issues like total operations, index scans per second, and rollbacks. If total operations are high, we’ll know at a glance if our database resources can handle the current workload intensity. If our index scans per second drop, this can suggest we’re not using indexes efficiently. Databases with a high number of rollbacks could be experiencing an increase in transaction failures or deadlocks. All of these things can lead to slow or unreliable performance for our users. Clicking into our database we see database-specific metrics: We can monitor index size for efficient resource optimation and right-size indexes. Dead row monitoring helps ensure efficient vacuuming to decrease table bloat and increase performance. It looks like we have 18 total operations per second, but 0 index scans per second, which might mean we aren’t indexing and could have some query performance inefficiencies.  Going into the PostgreSQL hosts navigator, we can view things like changes in operations per second, transactions, and disk usage to ensure our system can handle current workloads and maintain consistent performance:    We can also click into a specific host to view individual host metrics like how many transactions succeeded, failed, or were rolled back and the cache hits versus disk hits, both of which impact overall performance:  Moving between Infrastructure, APM, and Log Observer Our database monitoring journey will most likely start at the service level or with the applications they back, so let’s dig into query performance and how to view its impacts on overall application performance.  From within our PostgreSQL host navigator, if we select a specific host, we can view logs or related content in APM to view services that have a dependency on the currently selected host:  We can then jump to the Database Query Performance to view and analyze query time, latency, and errors to see which specific areas are impacting response time and user experience and where we might be able to optimize our query performance:  Closing this out, we can see our Service Map and where the current database sits so that we can investigate specific errors, traces, or related logs: We moved from Infrastructure to Application Performance Monitoring, but we could have just as easily started with our Service Map and began troubleshooting database performance issues from there using Database Query Performance, database requests/errors, or traces.  Wrap Up Monitoring key metrics from the databases that power our applications is critical to the performance and reliability that our users count on. Configuring the OpenTelemetry Collector to receive PostgreSQL telemetry data and export this data to a backend observability platform is an easy process that provides invaluable visibility into the databases that back our services. If you’d like to try exporting your Postgres data to Splunk Observability Cloud, try it free for 14 days!   Resources Automatic Discovery and Instrumentation of PostgreSQL with Splunk OpenTelemetry Collector Database Monitoring: Basics & Introduction OpenTelemetry Collector Configuring Receivers ... View more