This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. In this series, we go into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start with how to access the Splunk AI Assistant and identify unknown unknowns within a service environment, check out the first post in this series: Identifying Unknown Unknowns. In our second post, Analyzing and Troubleshooting in Real-Time, you can see how to use the AI Assistant to analyze trace errors and database query performance and gain insights into detectors and alerts. Our third post digs into how to leverage the Splunk AI Assistant to ensure compliance and optimize cost requirements. In this fourth post, we’ll explore how the Splunk AI Assistant can help explain unfamiliar or custom metrics and how it can provide real-time, contextual feedback to help analyze performance, identify optimizations, and troubleshoot faster for a reduced Mean Time to Resolution (MTTR).  Splunk AI Assistant Explains Metrics Within a company’s engineering organization, different teams may implement their own custom metrics specific to the services they own and manage. When troubleshooting an incident that spans services and teams, navigating these custom metrics and their meanings can slow down investigation and MTTR. Not only that, but engineers often utilize third-party tools that can be instrumented with custom metrics that don’t conform to internal standards or conventions. Interpreting these metrics can have a similar negative impact on troubleshooting time and MTTR. Let’s look at an example.  Our engineering team utilizes the data structure server, Redis, but I haven’t spent a lot of time working with Redis, so I’m unfamiliar with all of the metrics associated with our Redis instance. I’m on-call and dealing with an active incident, and I’m seeing a lot of Redis metrics pop up around cache hit rate percentage. I’m not sure what this means, but I can use the Splunk AI Assistant to gain more context and troubleshoot the issue.  From within Splunk Observability Cloud, I’ll navigate to Infrastructure and then select Datastores. I’ll scroll down to Redis and select Redis instances to get a detailed overview of my Redis instances:      From within the Redis instance view, I can open the AI Assistant and ask detailed questions about specific instance metrics, like this cache hit rate percentage. I can also ask it for contextual data to help me determine if the metric data is good or bad.  In my prompt, I’m going to specify the instance I’m investigating because, as you’ll remember from our previous post, Analyzing and Troubleshooting in Real-Time, the more context you provide to the AI Assistant, the more complete responses will be:  I’ve asked the AI Assistant to explain what the cache hit rate percentage metric is and how I can interpret it. The response returns analysis for this specific metric, showing that over the last four hours, the hit rate was 67%.  The interpretation of the contextual data in the response suggests that this is a moderate hit rate, indicating there is room for optimization. More details on suggestions for further optimization include reviewing the cache configuration, analyzing miss patterns, and application logic:  The response also provides important relevant Infrastructure attributes along with relevant dashboards and charts, complete with hyperlinks for further insight:  Wrap up To summarize the use case explored in this post, we’ve utilized the Splunk AI Assistant to explain unfamiliar metrics and provide real-time, contextual feedback, enabling us to troubleshoot quickly and reduce MTTR. In our next post, we’ll use the AI Assistant to streamline the onboarding process for new hires or new users of Splunk Observability Cloud to get everyone up to speed fast. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Previous post in the Splunk Observability Cloud’s AI Assistant Series: Auditing Compliance and Cost AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Explaining Metrics and Providing Feedback YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. Specifically, this series goes into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start with how to access the Splunk AI Assistant and identify unknown unknowns within a service environment, check out the first post in this series: Identifying Unknown Unknowns. In our second post, Analyzing and Troubleshooting in Real-Time, you can see how to use the AI Assistant to analyze trace errors and database query performance and gain insights into detectors and alerts. In this third post, we’ll learn how to leverage the Splunk AI Assistant to ensure we remain in compliance with our organization’s specific requirements. We’ll also see how the AI Assistant can analyze our infrastructure and identify areas that may be underutilized, which in turn will help us reduce costs. Compliance Organizations often implement code-level tagging requirements for compliance, security, cost, and operations. For example, it’s a requirement that all of the services in our Online Boutique environment set a tag value for the attribute tenant.level to easily distinguish between service tiers. This tenant.level attribute has three tag values – gold, bronze, and silver – as seen here in Tag Spotlight within Splunk Application Performance Monitoring:  We can use the AI Assistant to audit the data in our environment and ensure we are in compliance with this specific organizational requirement. We’ll ask the AI Assistant if any services in the Online Boutique environment aren’t using this required tenant.level tag. In the response, we can see that our Ad service is missing the required tenant.level tag:  Note: In the AI Assistant pane, if you have “Use current page filters” toggled on and you try to run this query from a specific page, like the paymentservice detailed view, you might not see the full results you expect. Querying from this page would set the context to just that of the paymentservice. If we want to interrogate our entire environment and all services instead, we would need to move up a level and execute the query from our services view that lists and sets the context as all services in our environment.  If we go to our Ad service in Splunk APM, we can verify that the data from it is in fact missing the required tenant.level tag: Cost Control Similar to what we saw in our last post about reducing alert noise and ecosystem hygiene, we can use the AI Assistant to analyze our infrastructure and help us identify areas that might not be efficiently utilized. This can help us capacity plan and reduce costs when allocating infrastructure.  We’ll have the AI Assistant analyze our Kubernetes infrastructure by prompting it to tell us whether it’s underutilized and identify nodes in Splunk Infrastructure Monitoring with all-time high CPU usage below 90% in our Online Boutique environment. The AI Assistant’s analysis identifies a node in our environment with a CPU utilization below 90%. The highest CPU utilization for this node is 73.3%, which is generally a great place to sit in terms of utilization, as it leaves room for CPU spikes. However, the average CPU utilization is generally lower than the maximum CPU utilization, and if this node is provisioned on a large instance type, like an EC2 m5.4xlarge, it’s likely over provisioned.  The response from the AI Assistant indicates that our node is operating efficiently. It also provides suggestions for potential optimizations, important attributes about the node, and links to Kubernetes charts and dashboards for continued monitoring:  Wrap up To summarize the use cases explored in this post, we’ve used the Splunk AI Assistant to audit compliance and cost by identifying expensive or underutilized infrastructure and discovering service configurations that may be out of compliance. In our next post, we’ll use the AI Assistant to explain specific metrics, contextualize data, and provide feedback. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Splunk Observability Cloud’s AI Assistant Series: Analyzing and Troubleshooting in Real-Time  AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Compliance and Cost Control YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at how to use the Splunk AI Assistant by digging into some practical, real-world, real-time examples. This series explores the specific use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics or providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code In our first post in this series, we saw how to access the Splunk AI Assistant and then explored how we could identify unknown unknowns within our service environment. The AI Assistant identified that our latest release could be causing high error rates and high latency. Before jumping into this second post, check out that first post: Identifying Unknown Unknowns.  This second post picks up where we left off and jumps into using the Splunk AI Assistant to: Perform deploy and release checks Investigate database query performance Interrogate active detectors and alerts Deploy and Release Checks We can use the AI Assistant to compare new code releases to previous versions. Let’s see this in action by asking the AI Assistant to compare the current latency of our Payment service to the prior version. We’ll ask it to compare Payment service performance for the current and previous versions in our Online Boutique environment and identify changes in behavior or error patterns for the latest release: The AI Assistant returns a comparison that shows the latest version of our service, version 350.10, has a much higher latency than the previous version, version 350.9:  Scrolling down to the error rate analysis, we can see version 350.10 has an error rate of over 30,000 errors, while version 350.9 has 0 errors. The analysis also shows that the common error for the latest version is “Invalid request with HTTP status code 401”: The AI Assistant also highlights observations and recommendations for further investigation and resolution of the issue:  The Observations section shows that the latency increase and error rate surge are significant and indicate “potential issues with the latest release.” The Recommendations include ideas for further investigation, analysis of code changes between versions, and rolling back to version 350.9 to maintain service stability. Analyze Trace Errors While investigating real-time issues, looking at specific trace data is helpful to get a deeper understanding of the root cause. We can use the AI Assistant to analyze trace errors for this troubleshooting in real-time use case. Here, we ask the AI Assistant to analyze trace errors specifically for our Payment service in our Online Boutique environment over the last 15 minutes: It’s important to note that being more specific with your AI Assistant queries – for example, specifying services, environments, time windows, and data types will help the assistant return more specific and useful responses.  In the response for our current prompt, we can see a list of trace errors for the Payment service with dynamic links to each of the traces: We can then ask for a deeper analysis of traces by prompting the AI Assistant with a specific trace ID:    The AI Assistant identifies the root cause of the errors as a 401 status code or unauthorized access to one of our third-party APIs: the ButtercupPayments service. The response includes the error message and error tags for the invalid request, and also an analysis of trace performance:  Scrolling down in the response, we can see the Performance Issues section that provides helpful information about the duration of the spans within our specific trace. The duration for this trace was relatively short, which hints at a pretty immediate rejection of the request. This section also highlights the service interaction, which explains how the data is requested. In this case, our Payment service is trying to request the Buttercup Payments API and failing due to this authorization issue: Looking at the recommendations, the AI Assistant suggests reviewing the authentication mechanisms for the Buttercup Payments API to ensure the credentials or tokens are valid, along with a couple of other recommendations:  Selecting the hyperlink at the end of this response will take us to the trace in Splunk APM so we can look at the data in detail and explore further: Database Query Performance Not only can we ask the AI Assistant to analyze traces and metrics related to our services, but we can also ask it to analyze database performance and investigate database queries. We can ask detailed questions like, “What are the top 5 worst performing database queries in the online boutique environment?” In the response to this question, we get a list of the database queries that have high latency, along with the number of times those queries were executed:  Detector and Alert Insights The Splunk AI Assistant can also provide insight into detectors and alerts. If we navigate to Detectors & SLOs in Splunk Observability Cloud and select critical alerts, we can see we have an active high error rate alert for our Payment service:  If we select that alert and scroll down in the alert window, we can get the Incident ID of this specific alert:  We can copy that Incident ID and then use that ID to ask the AI Assistant to explain this specific alert and tell us why it was triggered. In the response, the AI Assistant provides a summary of the alert details along with an explanation of the triggering conditions and why they were triggered:  We started investigating this specific alert by navigating to the Detectors & SLOs page view in Splunk Observability Cloud, but we could have also queried the AI Assistant to get information about active or frequently triggered detectors. The response to this query guides where to focus our attention: If these detectors are frequently triggered, it could result in service performance degradation and impact customer experience. The response to queries like this can also help us easily identify alerts that are triggered too frequently and are not actionable, which helps reduce alert noise.  Wrap up To summarize, this second post in our Splunk Observability Cloud’s AI Assistant in Action series focused on using the AI Assistant to perform deploy and release checks, analyze trace errors, investigate database query performance, and interrogate active detectors and alerts. This enabled quick insight into root cause analysis, service performance, and deployment impacts for quick issue resolution.  Stay tuned for our next post, in which we’ll use the AI Assistant to help manage organizational compliance and infrastructure costs. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Splunk Observability Cloud’s AI Assistant Series: Identifying Unknown Unknowns AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Deploy and Release Checks YouTube video Splunk Observability Hub ... View more

Agentic AI powers the Splunk AI Assistant within the Splunk Observability Cloud interface to help you quickly and easily tap into the health of your applications and infrastructure. Simply asking the AI Assistant your observability questions using natural language does the hard work of querying your environment and returning the relevant information you need to find root cause, triage, troubleshoot, and reduce Mean Time To Resolution (MTTR). The AI Assistant can even generate the SignalFlow you need to create custom charts and dashboards.  In this Splunk Observability Cloud’s AI Assistant in Action series, we will look at how to use the Splunk AI Assistant by digging into some practical, real-world, real-time examples. We’ll explore the specific use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining or providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code This first post will explain how to access the Splunk AI Assistant, and then we’ll learn how to use it to identify unknown unknowns within our service environment.  Accessing the Splunk AI Assistant You can access the Splunk AI Assistant from Splunk Observability Cloud by selecting the AI Assistant icon located in the upper right corner of the screen: Selecting this icon opens the interactive window to chat with the AI Assistant. Once the AI Assistant pane is open, you’ll see pre-populated prompts that are a great starting point if you’re new to the AI Assistant: Here’s an example response for the first prompt, "What can you help me with?”: As you can see, the AI Assistant can access metrics, traces, and logs in Splunk Observability Cloud. It can also help you analyze detectors, alerts, incidents, and draft SignalFlow.  Identifying Unknown Unknowns As an engineer, a great way to start the work day is to interrogate the AI Assistant with a question like, “What should I know about my environment?” Whether you’re back in the office after a weekend away, jumping into an on-call rotation, or simply catching up on a recent incident to help troubleshoot, using the AI Assistant to gain quick insight into unknown unknowns is a powerful resource.  Let’s ask a question about one of our environments. We’ll prompt the AI Assistant to tell us something we might not be aware of for one of the service environments we own, the Online Boutique. We’ll ask it to point out interesting or concerning things related to this environment: Looking at the response, we can see the AI Assistant performs analysis and then identifies that the Payment service has a high error rate. It also identifies common error tags like HTTP 401 status codes and high latency concerns: We can also see that the AI Assistant analyzed the upstream and downstream dependencies of the services in our environment: It also made some recommendations based on its findings. It hyperlinked to the services involved, so we can easily investigate those services in detail by moving over to Splunk Application Performance Monitoring for further troubleshooting. In this case, we’re hyperlinked to our Payment service and can explore real-time issues directly in Splunk APM:  My team owns the Payment service, and I know that code changes were recently deployed. A great next step would be to use the AI Assistant to interrogate our system and investigate if this latest release could be causing these high error rates and latency.  Wrap up To summarize, this first post in our Splunk Observability Cloud’s AI Assistant in Action series focused on using the Splunk AI Assistant to quickly identify unknown unknowns within our environment.  Stay tuned for our next post, in which we’ll use the AI Assistant to analyze and troubleshoot in real time, investigating error rates and latency and comparing deployments and releases. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Identifying Unknown Unknowns YouTube video Splunk Observability Hub ... View more

Web applications unfortunately present a target rich environment for security vulnerabilities and attacks. Adding in sometimes hundreds of third party dependencies and multiple services coordinating across a variety of technologies exponentially increases the impact and likelihood of these threats. Thankfully, Splunk AppDynamics (AppD) reduces the risk of application security exposure and additionally provides many features and capabilities to optimize hybrid and on-prem application performance. To name a few, AppD offers: Features to help identify application issues in real-time and tools to then prioritize how to tackle those issues based on business metrics and impacts  Root cause analysis from anywhere in your stack with AI-driven anomaly detection Business iQ Business performance monitoring to understand how application performance directly influences critical business metrics Digital Experience Monitoring (DEM) complete with Real User Monitoring (RUM) and Synthetic monitoring for insight into how end-users interact with our applications and observe availability and performance of web apps and APIs Network monitoring to quickly isolate network performance issues with complete visibility into owned and unowned network, ISP, API, SaaS, and third-party services Application security from inside your runtime environment with the detection of vulnerabilities, blocking of attacks, and insight into potential business impacts  In this post, we’re going to dig into the capabilities provided by Splunk AppDynamics with Cisco Secure Application and check out how they help enhance application security with real-time threat detection and prevention.  Splunk AppDynamics with Cisco Application Security With Cisco Secure Application, you can easily get complete security insight and protect your applications at runtime. Typically, security scans take place before an application is deployed to production, and scans continue to take place on a monthly or quarterly basis. But Cisco Secure Application continuously detects and blocks attacks. It also reveals the potential business impacts of these attacks and vulnerabilities so you can see and fix what matters to minimize business risk and easily prioritize issues.  Cisco Secure Application features are built into Splunk AppDynamics APM Agents for:  Java .NET Core Node.js Note: to monitor the application security, you must first enable security for the application using the Cisco Secure Application dashboard.  Overview The Cisco Secure Application home page provides a comprehensive and quick view of attacks and vulnerabilities.  Business transactions provide details about the vulnerability risk of a business transaction and the overview of vulnerabilities, attacks, and applications helps you quickly identify and respond to security incidents.  Applications The Applications page provides details on monitored nodes that are registered with Cisco Secure Application for the managed applications. Here, you’ll find a centralized view of all security-related details for your applications. This real-time data allows you to detect and respond to any applications that might not have security enabled for complete application coverage.  Stats on the nodes of managed applications are also available in this view. Nodes are categorized as: Active: the number of nodes that are actively communicating with the AppDynamics Controller Supported: the number registered with the Cisco Secure Application  Ready: the number of nodes that can be successfully registered Enabled: the total number of nodes that have security enabled for the applications Secured: the number of secured nodes Trend: the number of supported, enabled, secured, active, and ready nodes against the day of the month You can also quickly navigate to the flow map by clicking on the map icon to get a holistic view of the overall application performance.  Libraries The Libraries page breaks down details for existing libraries used within applications that require remediation.  Applications that use the corresponding library along with Tier (the application tier that is vulnerable because of its relationship to the corresponding library) are also easily visible. You can again click the map icon next to an application to view the application flow map in the AppDynamics dashboard.  This view also provides quick insight into:  The Highest Cisco Security Risk Score – an estimate of exploitation based on real-time events of the vulnerability in the wild A score based on the Highest Common Vulnerability Scoring System (CVSS)  Total vulnerabilities or the number of vulnerabilities based on severity  Remediation – the recommended version of the library that can be used for remediation Status of the vulnerable libraries Libraries that are categorized as "Critical" are potentially exploitable by a bad actor, which could disrupt your services or worse give access to customer data. These items should be prioritized for immediate triage. You can also click on a specific library to view vulnerabilities for that library:  Having this information easily visible means teams can quickly identify and prioritize the most critical vulnerabilities and threats. Impacted applications, version tracking, and remediation candidates provide insight into the impact and the actionable steps to take in order to keep the application secure. Vulnerabilities The Vulnerabilities page provides a complete view of the security of your services, focusing on the types and instances of vulnerabilities instead of on each library. It includes a real-time trend graph that shows the number of both fixed and open vulnerabilities.  Vulnerabilities are displayed by severity level, and the Severity Trend chart summarizes the trend of fixing open vulnerabilities. There’s also a Days Since First Detected chart showing the number of days the vulnerability is open versus the severity.  All of this detailed vulnerability information provides fast impact analysis and in-depth information on each vulnerability along with suggested fixes through remediation candidates.  Attacks The Attacks page is critical to providing information about attacks against your services. With real-time threat detection, you can get a high-level overview of security threats and the applications that they impact.  Attacks By Outcome represents the total number of attacks based on their states, and the Top Applications chart displays the top 10 applications based on attacks per application.  Attacks are also categorized for clear breakdown of the nature of the attacks. Attack types include SQL Injection, where an attacker is attempting to use unsanitized input to execute custom SQL statements to expose or manipulate private data, Remote Code Execution (RCE), where an attacker is attempting to gain access to a system and execute custom code, and others like:  DESERIAL LOG4J SSRF MALIP You can dig into specific attack details to view things like stack traces, source details, and the impacted business transaction: Note: If you have Configure permission for Cisco Secure Application, you can create or customize policies for vulnerabilities and attacks to specify which runtime behaviors to ignore, detect, or block. Cisco Secure Application Architecture and Installation How does this all work and how do you get started with Cisco Secure Application? According to the  Secure Application Architecture documentation: Install the supported APM Agent and then add the Cisco Secure Application license. The APM-managed application runs and the APM Agent retrieves the data to send to the Controller. The Cisco Secure Application service retrieves the application, tiers, and nodes data from the Controller. The APM Agent communicates with the Cisco Secure Application service to check if the security is enabled for the application. If the security is enabled, then the agent downloads the configuration along with the policies from the Cisco Secure Application service.  Based on the configured policies, the agent sends the security events to the Cisco Secure Application service. The service collects all the data, analyzes the application behavior, and then provides the analyzed data to the Cisco Secure Application dashboard.  Wrap Up Splunk AppDynamics with Cisco Secure Application provides real-time insight into the security health of your applications. Instead of waiting for a deploy to get visibility into vulnerabilities, Cisco Secure Application provides visibility at runtime and continuously detects and blocks attacks. This helps you quickly see and fix what matters to minimize business risk and easily prioritize what to work on.  If you’re ready to get started with Cisco Secure Application, you can first check out the requirements, install the agent that supports your environment, and then contact Splunk AppDynamics sales representative or email salesops@appdynamics.com to have a Cisco Secure Application license provisioned. More info can be found in the Application Security Monitoring Getting Started docs.    Resources Splunk Delivers Unified Security and Observability to Protect Applications Splunk AppDynamics Guided Tour Splunk AppDynamics Unveils New Security and Mobile Insights Splunk AppDynamics Application Security Monitoring ... View more

We’ve already explored a few topics around observability in a Kubernetes environment -- Common Failures in a Kubernetes Environment, how to Detect and Resolve Issues in a Kubernetes Environment, and the process of Integrating Kubernetes and Splunk Observability Cloud. But even though the process of setting up the OpenTelemetry Collector and exporting telemetry data to a backend observability platform like Splunk Observability Cloud can be completed in a few short steps, these initial setup steps don’t account for what happens when new workloads are spun up. Kubernetes is known for its dynamic nature, and observability needs to move right along with the ever-changing environment. In this post, we’ll see how monitoring Kubernetes workloads is currently done. Then we’ll check out how the new Kubernetes annotation-based discovery for the OpenTelemetry Collector makes it even easier. Monitoring Workloads: The Traditional Approach Monitoring dynamic workloads typically requires manual definitions of conditional configurations. Engineers define configuration “templates” based on environment conditions that adjust the monitoring configuration based on what’s automatically discovered in the environment. For example, to dynamically discover and monitor nginx pods that are deployed on the cluster, a receiver_creator definition is required:   This configuration is enabled when a pod is discovered via the Kubernetes API that exposes port 80 (HTTP port) and matches the nginx keyword. This approach works, but it requires a defined Collector configuration for each workload type. If new workload types are added to the environment, the configuration needs to be updated and redeployed, which can be time-consuming – especially if different teams are responsible for different aspects of the deployment. Additionally, it relies on the name of the pod, limiting flexibility to name workloads after what they actually do, making it harder to troubleshoot and diagnose problems. Monitoring Workloads: The New Approach The recently added Kubernetes annotation-based discovery for the OpenTelemetry Collector now allows for the dynamic configuration of monitoring workloads by adding annotations directly to their pods or namespaces. The K8s observer detects and reports information about objects appearing in the cluster to the Collector’s receiver_creator and automatically adjusts its configuration to monitor the annotated workloads. The annotations define the receiver that’s used to scrape telemetry data.  Here’s an example pod annotation that tells the OpenTelemetry receiver_creator to scrape metrics and enables discovery on the pod:    Finally, the discovery functionality must be enabled in the receiver_creator:   Once the deployment changes are applied and the Collector is restarted, you’re done! Adding a new nginx workload? No problem. Annotate your pods as shown above to dynamically configure monitoring for your new workload. It doesn’t matter what the pod is named and enables you to be more declarative about what is collected and how. Wrap up The amount of configuration maintenance required significantly decreases thanks to this new annotation-based discovery for Kubernetes and the OpenTelemetry Collector. Kubernetes observability becomes as dynamic as the environment it monitors, and configuration of dynamic workload discovery becomes a seamless experience. Already using the OpenTelemetry Collector in Kubernetes and want to try out this new feature? Want to export your automatically discovered Kubernetes workloads and their telemetry data to a unified observability platform? Check out Splunk Observability Cloud’s free 14-day trial. Resources Generate receiver configurations from provided Hints Kubernetes Monitoring and Alerting Made Easy with Splunk Observability Cloud and OpenTelemetry  Kubernetes Monitoring: The Ultimate Guide How to Monitor Kubernetes with Splunk Infrastructure Monitoring ... View more

Culture extends beyond work experience and coffee roast preferences on software engineering teams. Team culture also includes opinions on ticket creation/completion, merge request reviews, on-call rotations, and incident response. When it comes to incident response, a Don’t Repeat Incident (DRI) team or company culture can be hugely beneficial in maintaining a reliable code base, a positive user experience, and overall team happiness.  What is DRI? When an incident occurs, on-call team members are notified of triggered alerts. These individuals are often pulled into an incident response role to troubleshoot and resolve the incident. Traditionally, once the incident is resolved, the work of the on-call team members is over. Contrast this with a DRI approach: when the incident is successfully mitigated, the work is not over. Post-incident, work needs to be tasked out and prioritized to address any code changes or safeguards that need to be put in place to prevent the same incident from occurring again in the future – every incident results in the creation of one or more actionable tickets or issues. Sidenote: post-incident reviews (or postmortems) are also a great idea and serve as a retrospective around the incident to discuss why it occurred, how it could have been prevented, who was impacted, and how it will be prevented in the future.  When this whole process works, it helps create a proactive culture and reduces incident frequency. However, manual steps always leave room for error – a person forgets to create a ticket, a person forgets to follow up, and soon alerts for errors that have been seen before are firing yet again. Taking manual intervention out of the equation helps guarantee that tickets are created, added to the right repositories or boards, and prioritized to prevent incidents from repeating. How can technology help us with this? Cue webhooks. Webhooks Webhooks are a great way for systems to communicate when specific events occur. They can be used for a range of interactions – building out CI/CD pipelines, real-time data sharing like when sending confirmation emails after online order completion, initiating two-factor authentication at login, etc.   Because of their lightweight efficiency, many platforms support webhooks – Gmail, Slack, GitHub, Jira – the list is long. In this post, we’ll look at how Splunk Observability Cloud can help with the DRI practice described above to reduce incidents, increase code resiliency, and remove the need for manual intervention with webhooks.  Splunk Observability Cloud & Webhooks Your environment is unique, and Splunk Observability Cloud provides many integrations out-of-the-box (Slack, Jira, ServiceNow) that may suit your incident response needs. However, in situations where your environment needs aren’t met, custom webhook integrations have your back.  Say we keep track of our product’s code issues using GitHub issues. We can create a webhook integration in Splunk Observability Cloud to track active incidents by automatically opening an issue anytime an alert fires. Again, this helps ensure that human eyes land on the root cause and mitigation work is identified and recorded in an issue to make sure the incident does not repeat.   Let’s look at how we can go about setting up a GitHub webhook in Splunk Observability Cloud.  Configure a GitHub Webhook in Splunk Observability Cloud First, we’ll navigate to Data Management in Splunk Observability Cloud and search for the Webhook integration:    We can follow along with the guided setup and configure our GitHub connection:    Select Next to customize the auto-populated payload (but first, notice how much data you can send and act on in the remote system):    We’ll update the fields to match those required by the GitHub API to create a new issue and use the messageTitle and description variables to populate our issue:  After selecting Next we can review and save our webhook:  With our GitHub webhook saved, we next need to add it as a notification recipient on the desired detectors.  Note: to add a webhook as a detector recipient, you must have administrator access.  We can configure webhooks by editing the detector of choice:  We can add Alert recipients and select Webhook:   Then select our newly created GitHub Issue Webhook as the recipient and activate our updated alert notification:    Note: our GitHub webhook hits the GitHub REST API to create issues and these are scoped to the repository specified in the request URL – in our case, the Worms in Space repository. That means we would want to make sure the detector we attach this webhook to is related to that code repository specified in the URL. Thankfully, when we create detectors and alert rules, we can easily scope them to specific services.  That’s it! When an alert rule is triggered (and in this case when an alert is cleared or resolved), we’ll see a new issue automatically appear in our repo’s GitHub issues:  This is a simple example, but you can imagine how you could integrate this with a CI/CD system, for example, to monitor a critical metric, fires off a webhook that runs a script to determine if there’s been a recent release, and rolls back the release if the metric goes out of bounds. Exact examples vary a lot because there are so many systems out there, but the possibilities are really endless. Wrap up Now that our alerts create issues in our GitHub repository, we can prioritize and resolve their root cause and eliminate repeat incidents. Rather than being pulled away by reoccurring alerts, our focus can stay on delivering high-priority code, our applications can stay resilient, and our users can stay happy.  Want to accomplish something similar with Jira? Check out the Splunk Observability Cloud built-in Jira integration to easily connect Jira projects and create issues based on Splunk Observability Cloud alerts. Interested in building out a custom webhook? That works too! Once you’ve built a custom webhook, follow the same steps above so it can listen for and receive Splunk Observability Cloud alert notifications.  Don’t yet have Splunk Observability Cloud? Try it free for 14 days! Resources Send alert notifications to a webhook using Splunk Observability Cloud Send alert notifications to Jira using Splunk Observability Cloud Send alert notifications to services using Splunk Observability Cloud ... View more

Logs and Metrics Logs are generally unstructured text or structured events emitted by applications and written to files. Logs are essential to system visibility, especially when investigating unpredictable states of the system. Logs (ideally) give you data on exactly why something isn’t working. However, a logs-only approach becomes slow, expensive, and unsustainable as systems and the data they produce grow.   Metrics are data points that represent numbers, counters, or gauges that provide visibility into a system — error count, average CPU utilization percentage, throughput, etc. combined with a time stamp. Metrics are great for quick and reliable big-picture insight into system performance and can help teams quickly identify predictable system error states. But while out-of-the-norm metrics might indicate a problem within an application environment, it’s hard to pinpoint the root cause by looking at metrics alone.  Using a combination of logs and metrics provides many of the details needed to troubleshoot issues. So if we need both logs and metrics, what’s all this business about converting logs to metrics?  Why convert logs to metrics? In many systems, including Splunk, metric data points are stored in a format that provides faster search performance and more efficient data storage. This means log data that would be better represented by high-level, quantitative metrics rather than detailed text can be converted to metrics to help set alerting thresholds, reduce storage costs, and decrease the time it takes to troubleshoot issues. Rather than searching through logs of HTTP status codes and trying to parse patterns or identify outliers from the log text, we can convert status codes to metrics and easily view anomalies or even alert on them.  So why not just start off reporting these metrics? Maybe you’re already logging system data but are unable to alter the code base to instrument metric data – or perhaps you’re using a  commercial off-the-shelf (COTS) application that only emits logs. Maybe you need a low-barrier-to-entry way to gain deeper insight into your system with better search performance and lower data storage costs. Maybe certain teams need access to specific metric data but aren’t permitted to access the full details produced in logs. No matter the reason, pulling metric data from logs provides a higher-level system overview, a faster way to query data, and the ability to set alert thresholds on that metric data.  Convert logs to metrics with the OpenTelemetry Collector So how do we convert logs to metrics? Many observability platforms provide in-app ways to convert logs to metrics, so depending on which backend provider you use, the process can look different. In Splunk Enterprise, for example, ingest-time log-to-metrics conversion can be accomplished through Splunk Web or through configuration files.  For a quick and easy vendor-agnostic way to start reporting the metrics you need from the logs you already have, the OpenTelemetry Collector can be installed and configured to convert logs to useful metric counts. The Collector can be thought of as a pipeline to ingest logs from your application, process and extract metrics, and then send the metrics to an observability backend. Using the OpenTelemetry Collector approach allows us to:  Preprocess logs and offload the logs to metrics conversion from logging backends to the Collector  Send our metrics to locations outside of logging backends (like Splunk Observability Cloud) Reduce the need to store certain logs in logging platforms – instead, create the metrics and drop the logs or send them to low-cost archives Set our application up for future standardization by laying the groundwork for using OpenTelemetry Let’s take a look at how to do this.  Configuring the OpenTelemetry Collector Since we’re working with Splunk, we’ve installed the Splunk Distribution of the OpenTelemetry Collector. If you want to follow along and install the Collector yourself, you can work through the steps in the Splunk documentation to get started with the Splunk Distribution of the OpenTelemetry Collector.   With the Collector installed, we can configure the otel-collector-config.yaml file. First, we’ll update the receivers block to ingest our application’s log data:  This receiver sends our log data into the Collector just like it would to an HTTP Event Collector so no application code changes are required.  Next, we’ll need a connector. Connectors act as both an exporter and a receiver – they both consume and emit data. We’ll use a connector to consume our log data and then emit that data as a metric: The count connector is used to create a http_500_errors metric that counts any log events where the status code is equal to 500. The defined attributes will additionally create any dimensions we’d like to include on this count metric.  Note: we could also use the sum connector to convert our logs to sum attribute values. The connector you use depends on the type of metric you need and what you want the metric to represent. Jeremy Hicks wrote a super great, in-depth blog post on this called Introduction to the OpenTelemetry Sum Connector. Here’s a quick example of what using the sum connector would look like:  We’re going to move forward with the count connector example. We’ll want to export this new count metric to Splunk Observability Cloud, so we’ll also include an exporter for that:  If we wanted to send our logs to an HTTP Event Collector, we could also add another exporter here. In the future, we might even want to exclude exporting the logs we convert to metrics by including a filter processor here in our configuration.  Finally, we need to add these to the service pipelines: Here, we’ve configured our pipeline to receive logs from Splunk Enterprise, receive metrics from our count/http-error-500 connector, and export metrics to our otlp endpoint, aka Splunk Observability Cloud.  We can then save this configuration, start/restart the OpenTelemetry Collector, and search for our new metrics from the Metric Finder in Splunk Observability Cloud:  Wrap up Converting logs to metrics gives us a high-level view of our system and allows us to create dashboards, charts, and detectors to proactively detect and alert on anomalies. We can then use this information to dig into our logs for deep insight into the exact lines of code that are failing for super-fast issue resolution.  If you’re ready to convert your logs to metrics and visualize your metric data in an observability platform with unified, full-system visibility, start with a Splunk Observability Cloud 14-day free trial.  Resources Logs vs Metrics: Pros, Cons & When to Use Which From Webhook to Log and Log to Metric: OpenTelemetry Magic! Set up ingest-time log-to-metrics conversion in Splunk Web ... View more

Agent Saturation What and Whys In application performance monitoring, saturation is defined as the total load on a system or how much of a given resource is consumed at a time. If saturation is at 100%, your system is running at 100% capacity. This is generally a bad thing. Agent saturation is a similar concept. It represents the percentage of available system resources currently being monitored by an observability agent. 100% agent saturation means 100% of available resources are instrumented with observability agents, which is a great thing. When it comes to observability practices, agent saturation is how well a system is instrumented and can be represented by:  ( instrumented resources / total resources ) x 100 Because greater visibility into system health and performance means proactive detection of issues, improved user experience, more efficient troubleshooting, decreased downtime, and countless other pluses, 100% agent saturation is the ultimate goal.  So why doesn’t everyone get to 100% agent saturation for full system observability and magical unicorn application visibility status? It’s challenging! Setting up observability agents across distributed applications and environments takes time. In ephemeral, dynamic systems already integrated into existing solutions, it can just be too much of a lift.  But the good news is that if you’re already using Splunk (maybe for logging and security) there are quick and easy ways to improve your system observability. In this post, we’re going to look at how to leverage the Splunk Add-on for OpenTelemetry Collector to gain a quick win when it comes to improving agent saturation.  Improve Agent Saturation with the Splunk Add-on for OpenTelemetry Collector For Splunk Enterprise or Splunk Cloud Platform customers who ingest logs using universal forwarders, you can quickly improve agent saturation and deploy, update, and configure OpenTelemetry Collector agents in the same way you do any of your other technology add-ons (TAs). The Splunk Add-on for OpenTelemetry Collector leverages your existing Splunk Platform and Splunk Cloud deployment mechanisms (specifically the universal forwarder and the deployment server) to deploy the OpenTelemetry Collector and its capabilities for increased visibility into your system from Splunk Observability Cloud. The add-on is a version of the Splunk Distribution of the OpenTelemetry Collector that simplifies configuration, management, and data collection of metrics and traces. This means OpenTelemetry instrumentation will out-of-the-box exist anywhere the universal forwarder is present for logs and security use cases, making it easier to instrument systems quickly and gain visibility into telemetry data from within Splunk Observability Cloud. This comprehensive system coverage also comes with out-of-the-box Collector content and configuration with Splunk-specific metadata and optimizations (like batching, compression, and efficient exporting), all preconfigured. This means that you can get answers using observability data faster, saving you time and effort. Prerequisites for using the Splunk Add-on for OpenTelemetry Collector include:  Splunk Universal Forwarder (version 8.x or 9.x on Windows or Linux) Splunk Observability Cloud Splunk Enterprise or Splunk Cloud or deployment server as forwarders (Optional) Install the deployment server if you plan to use it to push the Collector to multiple hosts Getting started with the Splunk Add-on for OpenTelemetry Collector The Splunk Add-on for OpenTelemetry Collector is available on Splunkbase similar to other TAs, and you can deploy it alongside universal forwarders using existing Splunk tools like the deployment server.  We have a Linux EC2 instance we’re going to be instrumenting, but we first need to download the Splunk Add-On for OpenTelemetry Collector from Splunkbase:  We’ll unzip the package and then create a local folder and copy over the config credential files: In Splunk Observability Cloud, we’ll get the access token and the realm for our Splunk Observability Cloud organization:  Your organization's realm can be found under your user’s organizations:  Next, we set these values in our /local/access_token file: We then need to make sure the Splunk Add-On for OpenTelemetry folder (Splunk_TA_otel) is in the deployments app folder on the deployment server instance: We’ll then move over to the deployment server UI in Splunk Enterprise to create the Splunk_TA_otel server class and add the relevant hosts along with the Splunk_TA_otel app. Once the TA is installed make sure you check both Enable App and Restart Splunkd and select Save: That’s it! If we now navigate to Splunk Observability Cloud, we’ll see the telemetry data flowing in from our EC2 instance:  Wrap up Increasing agent saturation and improving observability for comprehensive system insight can be quick and easy. Not sure how you’re currently doing in terms of agent saturation? Check out our Measuring & Improving Observability-as-a-Service blog post to learn how to set KPIs on agent saturation. Ready to improve your agent saturation? Sign up for a Splunk Observability Cloud 14-day free trial, integrate the Splunk Add-on for OpenTelemetry Collector, and start on your journey to 100% agent saturation.  Resources Splunk Add-On for OpenTelemetry Collector Differences between the OpenTelemetry Collector and the Splunk Add-on for OpenTelemetry Collector Get started with Splunk Observability Cloud ... View more

Introduction In observability monitoring, static thresholds are used to monitor fixed, known values within application environments. If a signal goes above or below a static threshold or is within or outside a specified range, an alert will fire. Static thresholds are quick to configure and can provide helpful insight into system stats, but there are downsides to using them too frequently. There’s a time and a place for static thresholds, and in this post, we’ll look at when to use static thresholds, when not to use them, and alternatives to static thresholds.  When to use static thresholds Static thresholds work well for situations where there are “known knowns” – the predictable cases that you can anticipate – and when there’s a static range of “good” and “bad” values. Here’s an example of a CPU utilization detector with an alert rule that triggers when CPU is above 90% for a 5 minute duration:    As a side note, adding durations on such alert rules is important to avoid over-alerting on transient spikes. Without a set duration, every CPU spike over 90% would trigger an alert, as we can see when we try to configure the same condition without a duration:  The estimated alert count for this alert rule is 11 alerts in 1 hour – aka too much alert noise.  Boolean conditions, like monitoring synthetic test failures, are a great case for static thresholds. Alerting when a synthetic check fails indicates issues with site availability that should be addressed immediately. This detector alerts on a synthetic test when uptime drops below the 90% static threshold:    You can also use static thresholds to monitor Service Level Objectives/Service Level Agreements/Service Level Indicators (SLO/SLA/SLI). If you have an SLO of 99.9% uptime, you’ll want to be alerted anytime your availability approaches that threshold. Here’s an SLO detector that alerts if latency passes the 99.99999% threshold:    While configuring static thresholds on SLOs is possible, within Splunk Observability Cloud it’s recommended to manage SLO alerts using error budgets.  Static thresholds are most appropriate when working with fixed and critical metrics with clear failure conditions, e.g. HTTP error response codes spiking, response time increasing for a certain period of time, error rates above a certain percentage for a certain amount of time. If you’re just starting out on your observability journey, using static thresholds is also a great way to capture baseline metrics and gain insight into trends so you can fine-tune and adjust your detectors and alerts.  When not to use static thresholds As we saw in the CPU detector above, alerting on static thresholds can lead to a lot of alert noise if not used correctly. To create good detectors, we need to alert on actionable signals based on symptoms, not causes (we have a whole post on How to Create Good Detectors).  In dynamic system environments that include autoscaling and fluctuating traffic, static thresholds might not indicate actual system failures. Setting static thresholds on pod CPU in a Kubernetes environment, for example, could indicate increased load, but might not indicate a problem – pods could autoscale to handle the increase just fine.  Note: monitoring the CPU static threshold combined with pod queue length could provide the additional context needed to create an actionable detector.  When there are periods of traffic spikes – Black Friday for e-commerce sites, lunchtime for a food delivery site – setting static thresholds can lead to an increase in false alarms. Applications with such variable traffic fluctuations might not benefit from alerting on static thresholds alone.  Dynamic resource allocation and variable usage patterns can make using static thresholds in isolation tricky, but thankfully, alternative approaches can help. Alternatives to static thresholds For the situations mentioned above, along with those where we might not know all of our application’s failure states (unknown-unknowns), alternative detector thresholding or a hybrid approach can work best. Alternatives include but definitely aren’t limited to:  Ratio-based thresholds – instead of alerting on 500 MB of memory used, use a threshold of 80% of total memory for a specific duration of time Combining static thresholds with additional context – high CPU + pod queuing or error rate or latency  Sudden change detection – alerts on sudden changes to specified conditions like number of logins, response time, etc.  Historical anomaly detection to baseline environments and alert on deviations from trends – alerting on latency that deviates from historical trends Combining out-of-the-box approaches with custom thresholds can help you build a resilient monitoring solution to keep your applications running smoothly and your users happy.  Wrap up  For predictable, critical metrics static thresholds are a great alerting option. For situations when static thresholds aren’t appropriate, there are thankfully additional solutions that can help. To start, check out Splunk Observability Cloud’s out-of-the-box alert conditions here and explore what might work best for your unique application environment. Don’t yet have Splunk Observability Cloud? Try it free for 14 days!  ... View more

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth analysis on your incoming telemetry data. SignalFlow is the computational backbone that powers all charts and detectors in Splunk Observability Cloud, but the statistical computation engine also allows you to write custom programs in the SignalFlow programming language (modeled after and similar to Python). Writing SignalFlow programs definitely isn’t required for a complete observability practice, but if you want to perform advanced computations on your Splunk Observability Cloud data, SignalFlow is your friend. In this post, we’ll explore the whys and hows of SignalFlow so you can run these computations, aggregations, and transformations on your data and stream the results to detectors and charts for custom and in-depth observability analytics. Why use SignalFlow? Most Splunk Observability Cloud use cases don’t require complex computations. Out-of-the-box charts and detectors make building a complete observability solution easy. But sometimes there is a need for more detailed and advanced insight. SignalFlow can be used to: Define custom behavior or conditions for fine-tuned control over your monitoring so you can tailor your charts and detectors to your specific needs Aggregate metrics from different applications, cloud providers, or environments to unify data Troubleshoot by correlating metrics across many different sources – using SignalFlow during an incident helps provide deep, real-time investigation into root cause Detect trends over time or compare historical data to increase resiliency, reduce downtime, and capacity plan – i.e. correlate resources with user activity over time to optimize resource allocation Stream metric data to background analytics jobs – i.e. execute computations across a population over time Create reports or visualizations in third-party UIs so you can stream data out of Splunk Observability Cloud – i.e. in a service provider use case you can stream data to your own UIs and expose it to your customers Correlate business metrics with infrastructure and/or application metrics to understand how performance impacts customers – i.e. how does latency impact customer renewal rates There are many reasons why you might need to tap into SignalFlow; generally, if you need customized analytics, SignalFlow is the answer. So let’s see how it works! How to use SignalFlow You can define SignalFlow queries directly in the Splunk Observability Cloud UI or programmatically using the SignalFlow API. If you open up or create a chart in the UI, you’ll see the chart builder view:  If you select View SignalFlow, you can dive right into the SignalFlow that powers the chart and use it as a template for additional programs:  The same is true for detectors. If you open up a detector, you can select the kebab icon to Show SignalFlow:    SignalFlow programs outside of the Splunk Observability Cloud UI typically live within code configurations for detectors and dashboards (see our Observability as Code post). When you create a chart or detector using the API, you can specify a SignalFlow program as part of the request. Here’s an example of defining a detector using Terraform, where the program_text is the SignalFlow program:  You can also use the Signalflow API to run programs in the background and receive the results asynchronously in your client. Let’s take a look at some SignalFlow functions and methods we can use to build out charts and detectors.  SignalFlow Functions and Methods in Charts Most SignalFlow programs begin with a data() block. The data function is used to query data and is the main way to create stream objects, which are similar to a time-parameterized NumPy array or pandas DataFrame. Queries can run against both real-time incoming streams and historical data from the systems you monitor. In SignalFlow, you specify streams as queries that return data. Here’s a template for the data function:  We can expand on the data function in many ways. For example, here’s what it would look like to query for the CPU utilization metric and filter by host: We can also add/chain methods or functions to our data block. Here are examples of using the mean method to look at mean CPU utilization, mean CPU utilization by Kubernetes cluster, and mean CPU utilization over the last hour:  Operations like mean, variance, percentile, exclude, ewma, timeshift, rate of change, standard deviation, map(lambda), and others are available as methods on numerical streams. Here’s an example where our data stream, signal, is the CPU utilization with a filter of host, and we can add functions to timeshift by a week and two weeks, and then find the max value:  Comparing the max CPU utilization for two separate time series can’t actually be accomplished using the chart plot editor in the Splunk Observability Cloud UI, so this is an instance of where using SignalFlow is necessary.  To actually output these stream results to a chart, we need to call the publish() method:  We’ve now built out a chart using SignalFlow 🎉! We can also do this with detectors – read on. SignalFlow Functions and Methods in Detectors Detectors evaluate conditions involving one or more streams, and typically compare streams over periods of time – i.e. disk utilization is greater than 80% for 90% of the last 10 minutes. When building detectors using SignalFlow, we still start with our data streams, and then transform our data streams using boolean logic:  Note: when setting static thresholds in the UI, thresholds can only be greater than or less than. But as you can see with SignalFlow, we can specify greater than or equal to static thresholds.  We can use these when statements on their own or combined with and, or, not statements to publish our alert conditions and build out our detectors:  The detect streams in this example are similar to data streams. Detect streams turn our boolean statement – when our signal is greater than 90 for 1 minute – into an event stream. When this statement is evaluated as true, an event will fire and be published to an event stream. This is what triggers an alert.  Note: event streams are evaluated and published in real time as metrics are ingested, enabling you to find problems faster and speed up your MTTD. Every publish method call in a SignalFlow detect statement corresponds to a rule on the Alert Rules tab in the Splunk Observability Cloud UI. The label inside the publish block is displayed next to the number of active alerts in the Alert Rules tab:  You can create your detectors using the SignalFlow API, but if you want to use SignalFlow to build detectors directly in the Splunk Observability Cloud detector UI, you can append /#/detector/v2/new to your organization URL to do so:  Wrap up While working with SignalFlow is not required, it can help customize and advance your observability practice. A great place to start is editing the SignalFlow for existing charts and detectors in the Splunk Observability Cloud UI or using observability as code with SignalFlow programs. In no time, you’ll be building out SignalFlow program background jobs and streaming customized analytics to meet all your observability and business needs.  New to Splunk Observability Cloud? Try it free for 14 days!  Resources SignalFlow and analytics Analyze data using SignalFlow Up Close Monitoring with SignalFlow Intermediate to advanced SignalFlow ... View more

Observability protocols define the specifications or formats for collecting, encoding, transporting, and delivering telemetry data between telemetry data sources, intermediate sources like collectors, and telemetry backends like third-party observability platforms. The following table breaks down some of the most popular observability protocols, including OTLP, Prometheus, Jaeger, Zipkin, and others. Resources OpenTelemetry Protocol (OTLP) Prometheus StatsD OpenMetrics Jaeger Zipkin SysLog Fluent Forward  SNMP sFlow   ... View more

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to get the end-to-end insight we need to keep our applications healthy and our customers happy. In this post, we’ll explore how to integrate Amazon Elastic Kubernetes Service (EKS) with Splunk Observability Cloud so we can observe EKS alongside the rest of our application telemetry data.  Observability Metrics in AWS  The Amazon EKS management console and Amazon CloudWatch provide insight into EKS observability metrics. From the AWS management console, you can see things like cluster health and details about EKS resources deployed to your cluster. Node details are available within a selected cluster’s Resources tab:  You can dig into pod status, capacity, and pod details: Get an overview of node conditions like MemoryPressure, DiskPressure, etc.: You can even get insight into node events like node status:  For more detailed cluster health visibility, you can enable CloudWatch Observability from the EKS console. With Container Insights enabled, you get even deeper insight into cluster state and resource utilization:  Along with cluster, namespace, node, service, workload, pod, and container performance monitoring:  But if your infrastructure isn’t all in AWS and is spread across multiple platforms, navigating between observability tools, especially during a high-pressure incident, is not great. Instead, having one unified observability platform where you can view all these metrics and more reduces toil and time to incident resolution. End-to-end visibility unified in a central platform makes for a more resilient and efficient observability practice. So let’s look at how to integrate one such observability platform, Splunk Observability Cloud, with Amazon EKS.  Integrate AWS and Splunk Observability Cloud Splunk Observability Cloud provides a unified platform for troubleshooting and monitoring all application systems no matter where they live. Not only can you collect and store Amazon Cloudwatch Metrics data, but if pieces of your applications and infrastructure live outside of AWS, you can view that data right alongside your AWS data for a complete observability picture. You may have already integrated AWS with Splunk Observability Cloud through the Data Management section in Splunk Observability Cloud:  The integration wizard easily takes you through the process of preparing your AWS account:  And getting your AWS data flowing into Splunk Observability Cloud:  But for EKS, data is collected using the Splunk Distribution of the OpenTelemetry Collector, and even with AWS integrated with Splunk Observability Cloud, you’ll notice from the Available Integrations page that we still need to deploy the OpenTelemetry Collector to get our EKS data in:  So let’s install the Splunk Distribution of the OpenTelemetry Collector for Kubernetes.  Install the Splunk Distribution of the OpenTelemetry Collector We’ve gone through how to integrate Kubernetes and Splunk Observability Cloud before, and integrating Amazon EKS isn’t much different. When we follow along with the integration wizard, we just need to specify Amazon Web Services as the provider and Amazon EKS (or Amazon EKS / Fargate profiles) as the distribution:  We can connect to our EKS cluster and then follow along with the rest of the installation instructions. I’m using the AWS CLI in my terminal, but with Helm installed, you could also use AWS CloudShell. I first configured kubectl for my EKS cluster by updating my kubeconfig file:  And verified the connection:  I next ran the commands in the Splunk Observability Cloud installation instructions with splunk-otel-collector --version pinned to 0.111.0:  Once those steps were complete, I could then view my EKS telemetry data from within Splunk Observability Cloud:  In a previous post, we explored what it looks like to navigate Kubernetes data using Splunk Observability Cloud navigators to detect and resolve issues in a Kubernetes environment. Now that our EKS cluster is sending data to Splunk Observability Cloud, we can use all the same products and features within Splunk Observability Cloud to monitor our Amazon EKS environment.  From Infrastructure Monitoring we can view our Amazon EKS navigators:  We can get insight into all of our Kubernetes clusters:  Dive into the health of a specific cluster: And observe critical performance data around nodes, containers, daemonsets, deployments, namespaces, pods, replicasets, and workloads:  From these critical usage metrics, we can create detectors and alerts from within our navigators and they can live right alongside the detectors and alerts for the rest of our applications and infrastructure: With Amazon EKS now successfully integrated, we can use Splunk Observability Cloud to proactively monitor, detect, and alert on anomalies in our EKS environment right alongside the rest of our application and infrastructure telemetry data.  Wrap up Integrating with a third-party observability platform like Splunk Observability Cloud provides a unified observability solution for your applications and infrastructure. This helps with quick and easy incident detection and resolution without having to navigate between a bunch of different observability solutions.  Want to try integrating Amazon EKS with Splunk Observability Cloud? Try Splunk Observability Cloud free for 14 days!  Resources Get started with the Collector for Kubernetes Available Amazon Web Services integrations ... View more

At their best, detectors and the alerts they trigger notify teams when applications aren’t performing as expected and help quickly resolve errors to keep things up and running. At their worst, they cause an overwhelming amount of noise that fatigues our engineering teams, distracts from real application issues, and decreases the reliability of our products. In this post, we’ll look at ways we can skew toward the former to keep our detectors helpful, our engineers unpaged by unnecessary alerts, our applications performant, and our users happy.  What are detectors? Detectors set conditions that determine when to alert or notify engineering teams of application issues. These detectors monitor signals and trigger alerts when the signals meet specified rules. These rules are defined either by engineering teams or automatically by third-party observability platforms, like Splunk Observability Cloud’s AutoDetect alerts and detectors.  It can be tempting to create detectors around every possible static threshold that might signal that our system is in an undesirable state – CPU above a certain percentage, disk space low, throughput high, etc. However, creating too many detectors and setting off too many alerts doesn’t always lead to actionable solutions and causes alert fatigue. Creating the right detectors can help minimize false positives or negatives, reduce alert noise, and improve troubleshooting while reducing downtime and fatigue.   How to create good detectors It’s important to thoughtfully and intentionally create detectors to keep them as meaningful and helpful as possible. Detectors and the alerts they trigger should be:  Signals of an actual emergency or a signal that there’s about to be an emergency Actionable and based on symptoms, not causes Well-documented Built using Observability as Code Iterated upon  Let’s dig into each of these best practices.  Detectors should signal an actual emergency or impending emergency The Google SRE book defines alerting as, “Something is broken, and somebody needs to fix it right now! Or, something might break soon, so somebody should look soon.” Preferably, our system is self-healing and can autoscale, load balance, and retry to minimize human intervention. If these things can’t happen, and our users aren’t able to access our site, are experiencing slow load times, or are losing data, then our teams should be paged because there’s a real emergency that requires human intervention.  If the alert being fired doesn’t meet these criteria, it should perhaps be a chart living in a dashboard or a factor contributing to the service’s health score instead of an alert that could potentially wake engineers up in the middle of the night. For example, high throughput could lead to slower response times and a degraded user experience, but it might not (e.g. when CPU spikes precede auto-scaling events). Instead of creating a detector around high throughput itself, alerting on throughput thresholds and metrics surrounding latency or error rates would indicate an official or looming emergency.   At the end of the day, your application exists to serve a purpose. If that purpose isn’t being served or is in danger of not being served, alerts need to fire. If your infrastructure is running extremely hot, but the application is working, you can probably wait on alerting until business hours to look at additional capacity. But if users are being impacted, that’s a real emergency.  Detectors should be actionable Human intervention is expensive. It interrupts work on deliverables, distracts from other potential incidents, and can unnecessarily stress out and exhaust the humans involved – either by literally waking them from sleep or by fatigue from too many alerts. Therefore, when detectors and the alerts they trigger are created, they should define actionable states.  If a detector sets a static threshold on something like CPU and creates an alerting rule that triggers an alert when CPU spikes above 80%, the human responding to that page will go into an observability platform, see the CPU spike, and may not have much more information right off the bat. Are error rates spiking? Are response times slow? Are customers impacted? Is CPU spiking because of garbage collection or backup processing? Is any action necessary? Static thresholds like these don’t provide much context around end-user impact or troubleshooting. Instead, creating detectors and alerts around the symptoms or the things that impact user experience (think Service Level Objectives), means that there’s something wrong, there’s a real emergency, and a human needs to step in and take action.  Some examples of actionable symptoms include:  HTTP response codes like 500s or 404s Slow response time Users aren’t seeing expected content Private content is exposed to unauthorized users  Here’s a good example of a Splunk Observability Cloud detector that alerts when the error rate for the API Service is above 5% over a 5-minute time period:  Alerts would be triggered where the red triangles are shown.  Another example of a seemingly good detector is one that alerts when disk utilization percentage for a specific host goes above 80% for a specified amount of time:  However, this specific host initiates a data dump to S3 every 48 hours at 4 am EST, and this data dump causes a period of increased disk utilization. We can see in the alert settings that this alert is predicted to trigger in 48 hours when our S3 dump will occur, and we definitely don’t want to page the engineering team that manages this host at 4 am. We could adjust the singal resolution to a longer time period if needed, or we could create a muting rule that suppresses notifications for this specific condition:  The downside to this is that we could potentially miss a real incident, so we might also want to create a second detector around disk utilization percentage with a slightly higher threshold (that wouldn’t alert with our data dump spike) to monitor utilization during this time period.  Detectors should be well-documented Responding to an alert triggered by a detector should not require institutional knowledge and pulling multiple humans in to resolve the incident. There are times when this will happen, but detectors should be well-documented with runbooks for quick and easy guidance on troubleshooting steps.   Notifying the right people and setting the right escalation policies should also be considered in the configuration of detectors and alerts. For example, it doesn’t make sense to page the entire engineering team for every incident; it does make sense to page the code owners for the service experiencing issues. It doesn’t make sense to page leadership teams if a handful of customers are experiencing latency; it does make sense to page leadership teams if an entire product is offline.  Detectors should be built using Observability as Code Speaking of documentation… creating detectors and alerts as code provides a source-controlled documentation trail of why, when, and by whom the detector was created. This means that any new members joining the team will have background knowledge on the detectors and alerts that they will be on-call for. It also creates an audit trail when fine-tuning and adjusting thresholds, rules, notification channels, etc.   Using Observability as Code also makes it easier to standardize detectors and alerts between services, environments, and teams. This standardization means that all members of all teams have at least a baseline knowledge of the detectors and alerts and can potentially have greater context and ability to jump in and help during an incident.  Here’s an example of creating a Splunk Observability Cloud detector using Terraform:  Check out our previous post to learn more about deploying observability configuration via Terraform.  Detectors should be iterated upon  Detectors and alerts should work for our teams, not against them, and they should constantly evolve with our services to improve the reliability of our systems. To keep detectors and alerts actionable and relevant, it’s important to update, fine-tune, or even delete them when they’re no longer effective or helpful. If alerts are frequently firing but require no action, this is a sign to adjust thresholds or re-evaluate the need for the detector. If the actions taken by the incident response team could be easily automated, this is a sign to adjust or remove the detector. If anyone on your team uses the word “ignore” when describing a detector or alert, this is a sign to adjust or remove the detector. If team members are unable to troubleshoot an incident effectively based on the information they receive from a detector or alert, this is a sign to iterate on the detector.  Alerting detectors should never be the norm. If a detector is alerting, action should be required. Having a board full of “business as usual” alerting detectors quickly leads to false positives, false negatives, and ultimately hurts customer experience. Wrap up Detectors and alerts are critical to keeping our systems up and running, but too many detectors can work against that goal. Good detectors:  Signal real emergencies  Indicate an actual issue with user experience and provide our on-call teams with actionable insight Are well-documented Are built using source control Are continuously fine-tuned  These things help keep our detectors and alerts useful, our engineering teams focused, and our users happy.  Interested in creating some good detectors in Splunk Observability Cloud? Try out a free 14-day trial! Resources Google SRE book: Monitoring Distributed Systems How to Create Great Alerts Introduction to alerts and detectors in Splunk Observability Cloud Introduction to the Splunk Terraform Provider: Create a Detector in Splunk Observability Cloud ... View more

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to collect and monitor for database performance, health, and reliability. In this post, we’ll look at how to monitor MariaDB (and MySQL) so we can keep the data stores that back our business applications performant, healthy, and resilient.  How to get the metrics The first step to get metrics from MariaDB or MySQL (for the rest of this post, I’ll just mention MariaDB, but the same approach and receiver works for both) to Splunk Observability Cloud is to instrument the backend service(s) that are connected to your MariaDB(s). If you’re working with the Splunk Distribution of the OpenTelemetry Collector, you can follow the guided install docs. Here I’m using Docker Compose to set up my application, MariaDB service, and the Splunk Distribution of the OpenTelemetry Collector:   Next, we’ll need to configure the OpenTelemetry Collector with a receiver to collect telemetry data from our MariaDB and an exporter to export data to our backend observability platform. If you already have an OpenTelemetry Collector configuration file, you can add the following configurations to that file. Since I set up the Collector using Docker Compose, I needed to create an empty otel-collector-config.yaml file. The mysql receiver is the MariaDB-compatible Collector receiver, so we’ll add it under the receivers block along with our Splunk Observability Cloud exporter under the exporters block. Here’s what our complete configuration looks like:  As always, don’t forget to add the receivers and exporters to the service pipelines.  Done! We can now build, start, or restart our service and see our database metrics flow into our backend observability platform.  Visualizing the Data in Splunk Observability Cloud With our installation and configuration of the OpenTelemetry Collector complete, we can now visualize our data in our backend observability platform, Splunk Observability Cloud.  From within Application Performance Monitoring (APM), we can view all of our application services and get a comprehensive performance overview – including performance details around our MariaDB:  We can explore our Service Map to visualize our MariaDB instance and how it fits in with the other services in our application: And we can select our MariaDB instance to get deeper insight into performance metrics, requests, and errors:  If we scroll down in the Breakdown dropdown results on the right side of the screen, we can even get quick visibility into Database Query Performance, showing us how specific queries perform and seeing which queries are being executed:  Clicking into Database Query Performance takes us to a view that can be sorted by total query response time, queries in the 90th percentile of latency, or total requests so we can quickly isolate queries that might be impacting our services and our users: We can select specific queries from our Top Queries for more query detail, like the full query, requests & errors, latency, and query tags:  We can dig into specific traces related to high-latency database requests, letting us see how specific users were affected by database performance:  And also see right down into the span performance: And the db.statement for the trace: We can proactively use this information to further optimize our queries and improve user experience. You can also see that the database activity shows up in the overall trace waterfall, letting you get a full picture of how all components of the stack were involved in this transaction. But how can this help us in an incident? This is all helpful information and can guide us on our journeys to improve query performance and efficiency. But when our database connections fail, when our query errors spike, that’s when this data becomes critical to keeping our applications up and running.  When everything is running smoothly, our database over in APM might look something like this:  But when things start to fail, our Service Map highlights issues in red:  And we can dive into traces related to specific error spikes:  The stacktrace helps us get to the root cause of errors. In this case, we had an improperly specified connection string, and we can even see the exact line where an exception was thrown:  With quick, at-a-glance insight into service and database issues, we easily jumped into the code, restored our database connection issue, and got our service back up and running so our users could carry on with enjoying our application.  Wrap Up Monitoring the datastores that back our applications is critical for improved performance, resiliency, and user experience. We can easily configure the OpenTelemtry Collector to receive MariaDB telemetry data and export this data to a backend observability platform for visibility and proactive detection of anomalies that could impact end users. Want to try out exporting your MariaDB data to a backend observability platform? Try Splunk Observability Cloud free for 14 days!  Resources Database Monitoring: Basics & Introduction OpenTelemetry Collector Configuring Receivers Monitor Database Query Performance ... View more

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about integrating cloud-managed Kubernetes platforms? In this post, we’ll dig into how to monitor Google Kubernetes Engine (GKE) by integrating with Splunk Observability Cloud.  Wait, but why? GKE dashboards provide observability metrics around infrastructure and application health for clusters and workloads from within the Google Cloud Platform (GCP) itself. So why should we integrate GKE with an observability backend? Typically, not every piece of application infrastructure lives in the same cloud platform. In the middle of an incident when seconds matter, no one wants to navigate between platforms and remember where each discrete observability tool lives. Instead, having everything in one unified observability platform reduces toil and time to incident resolution. With end-to-end observability all in one place, root-cause analysis becomes a lot easier thanks to the ability to correlate issues impacting multiple parts of the stack. Third-party observability platforms provide advanced, flexible, and configurable dashboards, charts, and detectors along with support for application, service, and incident management integrations. Plus, depending on which platform you choose, you can avoid vendor lock-in by using an OpenTelemetry-native observability platform.  Observability Metrics in GKE We can monitor our applications running in Google Kubernetes Engine from GCP itself by configuring the Kubernetes Engine Monitoring setting on our cluster to System and workload wogging and monitoring. We can click into our deployed workload and see the CPU, memory, and disk utilization: And dig into the observability details around the health of our application – like container errors and restarts:  We can even go into Google Observability Monitoring and further explore GKE cluster performance: And build custom dashboards and charts:  These things are all great and useful, but again, to unify our observability tools and improve the time to incident resolution, how can we get all of this information into a single backend observability platform? Integrate GKE and Splunk Observability Cloud There are a couple of ways you can integrate GKE into Splunk Observability Cloud. We’ll first follow along with the Connect to Google Cloud Platform documentation.  Integrate Google Cloud Platform From within the GCP UI, we’ll go into the project we want to monitor and add a new Splunk Service Account under IAM & admin: Once the new service account is created, we can edit it to create a new service account key:  And then download the key as JSON:  Note: GCP Cloud Resource Manager API must be activated so Splunk Infrastructure Monitoring can use it to validate permissions on the service account key. Also, if you want to monitor multiple GCP projects, the above steps need to be repeated for each one. Next, we’ll navigate to Splunk Observability Cloud and add our Google Cloud Platform integration:  We can then import our newly created service account key: A benefit of integrating GCP in this way is that we can sync all supported GCP services automatically. However, for our purposes, we’ll optionally refine the GCP synced services to only include GKE:  Once saved, we’ll see Google Cloud Platform under our actively deployed integrations:  Depending on the polling interval, it might take a few minutes for our GCP metric data to populate within Splunk Observability Cloud. Integrate Google Kubernetes Engine  If you’re only interested in integrating GKE without additional GCP service integration (or if your security team won’t give you a service account), you can integrate with Splunk Observability Cloud via Helm chart. We can search for Google Kubernetes Engine in the list of available integrations and follow the guided Kubernetes Monitoring instructions: We’ll specify Google Cloud Platform as the provider and Google GKE as the distribution:  And then we’ll follow along with the installation instructions: In the GKE UI, activate a cloud shell, and then enter the above commands within the instance:  You’ll notice our helm-install splunk-otel-collector command failed because of a missing instrumentation.endpoint value. This issue results from the fact that the version of helm pre-installed in the Google Cloud Shell is out of date. To resolve the error, we need to upgrade the helm version to one supported for the current Kubernetes version and rerun the commands in the Splunk Observability Cloud installation instructions:  After this completes successfully, we can go back over to Splunk Observability Cloud and see our integrated telemetry data: From here, we can explore data like CPU and memory usage/utilization, resource capacity, container restarts, node state, and others right alongside the rest of our application and infrastructure data within Splunk Observability Cloud, even if other parts of the app run on-premises or on another public cloud provider.   Observability Metrics in Splunk Observability Cloud Now that our GKE integration is complete, we can use all of the available Splunk Observability Cloud products and features to monitor our GKE environment. From Infrastructure Monitoring we can view our Google Cloud Platform navigators:  And dive into the health of our clusters:  Dig into a specific cluster:  Monitor pod health and performance:  And view critical usage metrics:  We can create detectors and alerting rules from within our Navigators and manage them alongside the detectors for the rest of our applications and infrastructure: In a previous post, we looked in detail at how to use the Kubernetes Navigators to Detect and Resolve Issues in a Kubernetes Environment. With GKE now integrated with Splunk Observability Cloud, we can use these same tools to proactively monitor, detect, and alert on anomalies in our GKE environment.   Wrap up  We can monitor our GKE environment from within GCP itself, but integrating with a backend observability platform like Splunk Observability Cloud unifies our monitoring solution. With one unified observability platform, we can more easily detect incidents and resolve them faster without having to navigate between different observability tooling. Want to try integrating GKE with Splunk Observability Cloud for yourself? Try Splunk Observability Cloud free for 14 days! Resources Connect to Google Cloud Platform: Guided setup and other options Install the Collector for Kubernetes using Helm Splunk OpenTelemetry Collector for Kubernetes ... View more

We can’t guarantee the health of our services or a great user experience without data from our applications. Is CPU usage high? Are there an increased number of requests? Do we have too many Kubernetes nodes stuck in a NotReady state? These metrics and many others impact our services and our customers, but if we can’t see them, we can’t fix them. So, we build out an observability practice, instrument our services, collect all the metrics, export metrics to an observability backend, and quickly realize that our systems produce an overwhelming amount of data. We store metrics to understand trends, but data storage costs money so collecting and storing everything quickly becomes a budgetary problem. In the noise of all that data, it’s also difficult to identify which metrics are relevant in determining what is actually negatively impacting our services and users.  In a world where setting up a functional observability practice is as easy as installing the OpenTelemetry Collector configured with auto-instrumentation, there are different ways to manage metric pipelines so that metric collection is sustainable and cost-effective and provides real value to the reliability of our applications. In this post, we’ll look at how OpenTelemetry processors specifically can help manage data from within metric pipelines to avoid exporting and storing unhelpful data so you can focus on service reliability, faster troubleshooting, and lower observability costs. Managing Metric Data Volume Best Practices The following best practices help eliminate metric noise, reduce metric collection volume, and ensure helpful metrics are available and ready to support troubleshooting efforts.  Use OpenTelemetry Semantic conventions for metric names and attributes Collect metrics intentionally  Monitor the pipeline itself Optimize the pipeline and exporting processes Let’s take a look at each of these. OpenTelemetry Semantic Conventions Using OpenTelemetry metrics semantic conventions when naming metrics or metric attributes helps with data analysis and troubleshooting. Defining clear metric names and attributes also helps identify redundancies or commonalities between metrics. Without the use of semantic conventions, different engineering teams might use different names for the same metric, leading to metric redundancy and increased data volume. For example, metrics around total HTTP requests could be named: http_requests_total, total_http_requests, http_request_count, etc. With semantic conventions in place, these individual metrics can be consolidated into one single, shared metric like http.server.requests, which captures aggregated total requests and attributes like request method and endpoint. When metrics follow naming conventions, aggregations, filters, and transformations can more easily be applied to reduce the volume of metric data, reduce the cost of backend platform storage, and improve the effectiveness of observability practices.  Collect and Store Metrics Intentionally With semantically named metrics, it’s easier to identify those that provide value and rename or remove the ones that don’t, but how do you determine which metrics are and are not helpful? Here are some questions to consider: Could the metric be used for an actionable, high-priority alert?  Would the data reported by the metric create a meaningful dashboard? Is the individual metric meaningful? Or would an aggregation be more impactful?  It’s also important to note that not all metrics need or should be exported to backend observability platforms – not all data is relevant for troubleshooting or development purposes and doesn’t need to be readily available within observability backends. Cold data that won’t actively be used, like metrics necessary for compliance or audit purposes, can be exported to backend storage like Amazon S3 (perhaps even Glacier). This can lower storage costs and keep observability backends clear of metrics that aren’t immediately helpful in monitoring the resiliency and performance of applications.   Monitor the pipeline Monitoring the pipeline itself (e.g. Collector performance and resource limits) can help identify delays and/or constraints in processing or exporting. This ensures data integrity and quick insight into any issues with metric collection. It also provides insight into the performance and effectiveness of metric collection so you can iterate on which metrics you’re collecting and how you’re collecting them. Optimize the pipeline and exporting process Optimizing the pipeline collection and exporting processes ensures efficient data flow from collection to the backend platform so you can prevent bottlenecks and delays and successfully use the metrics you collect for performance monitoring and troubleshooting. OpenTelemetry Processors So how do you put these best practices into… practice? The OpenTelemetry Collector provides several processors that can be configured to transform data before it’s sent to observability platform backends. We can think of these processors more as pre-processors, taking many points of data and interpreting or condensing them into more meaningful information. Processors offer more control over metric collection so data can be reported in useful ways that reduce metric noise and storage costs.  Filter Processor Metric data can be included or excluded through configuration of the filter processor in the OpenTelemetry Collector configuration file. Any low-priority or unhelpful metrics, like those with invalid types or specified values, can be filtered out. Here’s an example that shows how to drop an HTTP healthcheck metric:  This metric doesn’t provide meaningful or actionable data around application performance or reliability, so to reduce metric volume and storage costs, we can drop it before exporting it to our backend observability platform. Attribute and Metric Transform Processors The OpenTelemetry attribute and metrics transform processors can be configured to modify and/or consolidate metrics. Their functionalities overlap a bit – you can add attributes or update attribute values using either processor. The metric transform processor provides more room for data manipulation, and the docs recommend that if you’re already using the metrics transform processor functionality, there’s no need to switch over to the attribute processor.  The metric transform processor can be used for renaming metrics so you can modify metric names or attributes while sticking to semantic conventions to reduce the number of discrete but related metrics, which are often billed separately in observability backends. For example, when using multiple cloud providers like Amazon Web Services (AWS) or Google Cloud Platform (GCP), each provider reports CPU utilization data under different metric names. Instead of reporting these metrics separately, they can be combined into a single metric name following semantic conventions to reduce cardinality and improve metric management. Here’s an example of updating AWS and GCP CPU utilization metrics to report under a single cloud.vm.cpu.utilization metric with attributes indicating the cloud provider and cloud service:  Group by Attributes Processor To organize metric data and more easily apply aggregations and transformations to specific groups of metrics, use the group by attributes processor. For example, if you’re collecting data from multiple services each running on multiple instances, the group by attribute processor can be configured to group metrics by service name as follows: Aggregations can be applied to sum or average metrics for each instance of each service. If service_a is running on instance_1, instance_2, and instance_3, we can use the group by attributes processor to combine these individual instance metrics into one single aggregated service_a metric. This reduces cardinality and data volume, while also making the metric data easier to troubleshoot.  Batch Processor While the batch processor doesn’t manipulate the raw metric data itself, it does contribute to an effective metrics pipeline by improving export performance. Effective exporting of our data means it gets to where it needs to go and is readily available for use when we need it. Batching metrics to compress data reduces the number of outgoing connections in order to improve exporting performance. It can easily be configured within the Collector configuration file by specifying batch under the processors block:  Or additional configuration options like batch size and timeout can be specific for more fine-grained control:  Memory Limiter Processor Like the batch processor, the memory limiter processor is related to the overall functionality of the metric pipeline. Using this processor ensures metric collection functions properly and data is collected, processed, and exported successfully. The memory limiter processor performs periodic checks of memory usage to prevent the Collector from running out of memory. If the Collector hits memory limits, it will start refusing data and lead to metric data loss. Here’s how to configure the memory limiter with a couple of the available options within the Collector configuration file: Wrap Up Observability isn’t about collecting all of the data. Collecting all the data is counterproductive to maintaining reliable systems that successfully support our customers. Instead, observability is about surfacing and analyzing actionable data. Managing metrics data volume at the point of collection with OpenTelemetry processors can help reduce the noise making it easier to detect anomalies and resolve issues faster. OpenTelemetry, a native part of Splunk Observability Cloud, provides built-in metric pipeline management and one unified observability backend platform for profiles, metrics, traces, and logs – no third-party pipeline management tools required. With Splunk Observability Cloud, you can also manage some metrics after ingestion with Metrics Pipeline Management (MPM). Interested in reducing metric volume, storage costs, and troubleshooting efficiency? Start a Splunk Observability Cloud 14-day free trial and adopt OpenTelemetry to tame your metric pipeline and to experience the benefits of one centrally-located backend observability platform (aka Splunk Observability Cloud). Using the power of Splunk Observability, watch your metric data flow in and your data storage costs go down (all while optimizing troubleshooting and reducing time to resolve incidents thanks to helpful and well-managed metric data).  Resources OpenTelemetry Metrics Transforming telemetry OpenTelemetry Collector Configuration Data Storage Costs Keeping You Up at Night? Meet Archived Metrics ... View more

If you’re beyond the first-weeks-of-a-startup stage, chances are your application’s architecture is pretty complex. Especially in CI/CD pipelines, things change on a near daily basis, so what existed in your codebase last week might not this week. In such fast-moving, large-scale environments, it would be impossible for development teams to manually keep track of and hold in their heads every line of code, every change, every new piece of functionality. So why would we expect the same for every piece of infrastructure or observability configuration?  Terraform is a popular Infrastructure as Code (IaC) tool created by Hashicorp that solves for this complexity challenge by getting the manual pieces out of our heads and into automation pipelines and version control. In this post, we’ll start with Terraform basics. We’ll walk through the benefits of using Terraform in general, then move into the “whys” behind incorporating Terraform into observability. Finally, we’ll take a look at how to implement the Splunk Terraform provider to get you up and running with Observability as Code.  Terraform: what is it & why use it? Rather than going into UIs or manually interacting with CLIs to deploy infrastructure like AWS EC2 instances, Kubernetes clusters, or even entire short-lived test environments, Terraform is a software tool that enables the provisioning and managing of infrastructure via code. This allows development teams to treat their infrastructure (and observability for infrastructure and applications) like any other piece of code they create, making it easy to maintain, update, and deploy additions or changes. Resources are defined in human-readable Terraform (*.tf) files that declaratively describe the desired infrastructure end-state. Plugins, known as providers, interact with cloud platforms and services to define pieces of infrastructure. There are currently over 1,000 providers in the Terraform Registry for managing resources on AWS, Azure, GCP, Kubernetes, Splunk, and many other platforms.  So why might you want to use Terraform? As we said, Terraform takes the tedious, manual, error-prone provisioning out of the infrastructure management process. Instead, with Terraform, teams can build, change, and manage infrastructure in a version-controlled, shared, and repeatable way all from a centralized location – a Terraform file. This file can be committed and stored alongside existing code in GitHub, GitLab, etc. for safe and collaborative infrastructure management. Additionally, multiple cloud platforms can be managed from within the same Terraform file, eliminating the need to manually move between different configuration interfaces. Terraform’s human-readable file structure makes it quick and easy to define and update infrastructure, and the state file that’s created with every deployment keeps track of existing infrastructure and changes so developers don’t have to.  It might sound like the process of using Terraform would be complicated, but it’s pretty straightforward. After installing Terraform, all you need to get started is a main.tf file and a few quick commands to initialize your configuration, view your infrastructure plan, and apply or deploy the defined infrastructure. Terraform & Observability How does observability relate to Terraform or Infrastructure as Code? Like infrastructure, observability configurations can be managed as code using Terraform. Rather than managing resources like dashboards, charts, detectors, or cloud platform integrations via observability platform UIs, they can be configured and deployed with Terraform. What does this bring to a team’s observability practice?  Standardization & Consistency Terraform managed resources can be packaged into modules making them reusable and easy to deploy. This also means that resources for different services and different environments can share configurations, making observing the end result in observability platforms more consistent. Consistency for resources like dashboards, charts, or alerts makes it easier to interpret data across services, products, or environments. During an incident, knowing where to find key metrics or being able to compare them across environments means faster insight into root cause and decreased time to incident resolution. True, this consistency could be accomplished by manually creating the same dashboards and charts across services and service environments, but this is tedious and leaves room for errors and configuration drift.  Collaboration & Improved Maintenance Managing observability configurations in code means version-controlled audit logs or documentation of changes to provide context around why resources were created or updated. Unlike with manual configurations where one person can go into the UI and adjust a setting, followed by another person subsequently reverting that setting, or even two people adjusting the same configuration at once, version-controlled configurations must be checked in and approved through the merge request process just like every other code change. Additionally, if updates don’t work out as expected, there’s a simple rollback process to revert changes.  Increased Speed & Scalability With Terraform, observability configurations are stored in one place – alongside the codebases they monitor. This means no scrambling to find one of hundreds of alerts in a UI to adjust its thresholds – instead, simply go into the codebase to tweak and redeploy. Changes can also be applied across multiple resources and multiple environments quickly, taking away the need to scroll through all resources, make adjustments, and then rinse and repeat for each additional service or environment. It might seem over-kill to create dashboards and charts via Terraform; however, in a world of microservices, where each microservice exists in multiple environments (e.g. dev, staging, production), deploying observability resources via code not only speeds up development, it speeds up issue detection and resolution.  Improved Security Rather than provisioning observability platform credentials to multiple team members, observability as code keeps things like API keys in one place – with the code – so they can be managed alongside other code secrets for improved security.  Splunk Terraform Provider Let’s see how easy it is to create observability resources using the Splunk Observability Cloud Terraform provider.  Note: If you don’t already have Terraform installed, start there. Terraform documentation is super thorough and includes a ton of great tutorials.  We’ve created a new main.tf file in our application’s root directory, but if you’re already using Terraform, you can add the same configuration additions to your existing Terraform file. Here’s an example of a Splunk Terraform provider configuration:  The first section is our required_providers block and declares the provider dependencies that we’ll be using, in our case the Splunk Observability Cloud aka signalfx provider. Think of the required_providers block like an import statement, while the following provider-specific block configures the signalfx provider.  Note: since our Splunk organization is located in us1 and not in the default of us0, we needed to specify the api_url.  We then define resource blocks. Each resource block describes the observability objects we want to create or update – dashboards, detectors, charts, etc. All available configurable resources can be found in the Splunk Observability Cloud provider docs under the Resources section. In our configuration, we’ve defined two resource blocks: a dashboard group, and a dashboard.  Now from this root directory, we can run the terraform init command to initialize the new Terraform infrastructure and install the Splunk Observability Cloud provider:  We can next run terraform plan to see the actions that will be taken once we apply our configuration: Two resources will be created, a dashboard group and a dashboard like we expect, so we can next run terraform apply to create these two resources:  It looks like our resources were successfully created, and we can head over to the Splunk Observability Cloud UI to confirm.  Going into our dashboards in Splunk Observability Cloud, we can see our newly created dashboard group and dashboard:  But this isn’t super useful yet because there’s no real information here that can help us on our observability journey. But that’s ok, to add new resources or update existing ones we can simply edit our main.tf file. We’ve added a chart resource to our Terraform file and updated our current dashboard to include the new chart:  We can then run terraform plan to again view the plan for the updated actions: That looks good, so we can apply the changes by running terraform apply and view our newly Terraform-created chart within our Terraform-created dashboard and dashboard group:  Wrap Up Whether you’re building out detectors and alerts, dashboards and charts, or integrating cloud services via Terraform, managing observability configurations as code can improve resource consistency, collaboration, and maintenance while increasing development and incident resolution speed. If you’re new to Terraform, check out some of the additional resources below. If you’re already using Terraform, try out observability as code and add some Splunk Observability Cloud resources to your Terraform configuration. Need Splunk Observability Cloud? We have a free 14-day trial! Resources Introduction to The Splunk Terraform Provider Infrastructure and Observability as Code Using the Splunk Observability Cloud Terraform Provider Managing Splunk Observability Cloud Teams As Code Terraform Documentation Splunk Observability Cloud provider ... View more

Behind every business-critical application, you’ll find databases. These behind-the-scenes stores power everything from login and checkout to content lookups and “likes,” so issues with slow queries, too many full table scans (or too few index scans), incorrectly configured indices, or resource exhaustion directly impact application reliability and user experience. Thankfully, we can capture key database metrics to expose such issues and ensure optimal performance, efficient troubleshooting, and the overall reliability of our applications.  In this post, we’ll explore monitoring the open-source relational database PostgreSQL. Postgres is widely used in enterprise applications for its scalability, extensibility, and support. It also collects and reports a huge amount of information about internal server activity with its statistics collector. We’ll harness these stats using the OpenTelemetry Collector and first focus on the database and infrastructure itself in Splunk Observability Cloud. Then we’ll see how everything connects to our application performance data. Which metrics matter and why Monitoring database metrics is critical to proactively identifying issues, performance optimizations, and database reliability, but with so many stats coming from the statistics collector, it can be difficult to determine what to focus on. How do we isolate what’s critical to monitor effectively? It can help to focus on operation-critical key metrics like those related to: Query performance (query throughput/latency, locks, query errors, index hit rate) Resource utilization (connections, CPU, memory, disk space, table/index size, disk I/O, cache hits) Database health (replication lag, deadlocks, rollbacks, autovacuum performance) Query Performance Slow, resource-intensive queries or queries with high throughput can decrease the response time of our applications and degrade user experience. To prevent things like slow page load time, we want to focus on metrics related to query time –  total response time, index scans per second, and database latency. These metrics will indicate if our database has the right or wrong indexes, absent indexes, if our tables are fragmented, or have too many locks, etc.    Resource Utilization Exceeding resource thresholds can halt application operations altogether. If total active connections are too high resources might be exhausted, and users might not be able to interact with our application at all. Monitoring resource usage like CPU, memory, and table/index size can keep our databases up and running, while also allowing for accurate capacity planning and optimal user experience.  Database health Things like a high rollback to commit rate can indicate user experience issues, for example, users might be unable to complete product checkout on an e-commerce site. An increase in the number of dead rows can lead to degraded query performance or resource exhaustion with similar effects. Proactively monitoring these metrics helps easily identify inefficiencies, eliminate bottlenecks, reduce database bloat, and ultimately improve user experience.   How to get the metrics So how do we get these metrics from PostgreSQL to the OpenTelemetry Collector? The first step is installing the OpenTelemetry Collector. If you’re working with the Splunk Distribution of the OpenTelemetry Collector, you can follow the guided install docs. I’m using Docker Compose to set up my application, Postgres service, and OpenTelemetry Collector, so here’s how I added the Splunk Distribution of the OTel Collector:  If you already have your OpenTelemetry Collector configuration file ready to edit, you can proceed to add a PostgreSQL receiver to the receivers block so you can start collecting telemetry data from Postgres. Because I set up the Collector with Docker Compose, I manually created my Collector configuration file (otel-collector-config.yaml). Here’s the PostgreSQL receiver I added to my Collector config:  Note: generally, your database and microservices would be behind network and API security layers so your databases and services would talk to each other unencrypted, which is why I have tls set to insecure: true. If your database requires an authenticated connection, you’ll need to supply a certificate similar to what’s shown in the documentation’s sample configuration.  I’m also exporting data for my application to my Splunk Observability Cloud backend, so I’ve added an exporter for that and added both my new receiver and new exporter to my metrics pipeline:  If you’re not using the Splunk Distribution of the OpenTelemetry Collector or not exporting data to Splunk Observability Cloud, configuring the PostgreSQL receiver block will still follow the example shown, but you’ll need to configure a different exporter and add it to the metrics pipeline.  That’s it! Now either build, start, or restart your service (I did a docker compose up --build) and watch your database metrics flow into your backend observability platform of choice.  Note: If you’re working with a complex service architecture and the Splunk Distribution of the OpenTelemetry Collector, you might want to consider using automatic discovery. This allows the Collector to automatically detect and instrument services and their data sources. Depending on your environment and Collector installation method, you can follow the appropriate docs (Linux, Windows, Kubernetes) to deploy the Collector with automatic discovery. How to see the data in Splunk Observability Cloud Now that we’re collecting Postgres data, let’s jump over to Splunk Observability Cloud Infrastructure to visualize our telemetry data. We can select the Datastores section and open up either our PostgreSQL databases for database-level metrics or PostgreSQL hosts for metrics related to the infrastructure hosting your PostgreSQL database(s): Going into the PostgreSQL databases navigator, we can see the metrics related to all of our databases: Here we see those key metrics that can hint at performance issues like total operations, index scans per second, and rollbacks. If total operations are high, we’ll know at a glance if our database resources can handle the current workload intensity. If our index scans per second drop, this can suggest we’re not using indexes efficiently. Databases with a high number of rollbacks could be experiencing an increase in transaction failures or deadlocks. All of these things can lead to slow or unreliable performance for our users. Clicking into our database we see database-specific metrics: We can monitor index size for efficient resource optimation and right-size indexes. Dead row monitoring helps ensure efficient vacuuming to decrease table bloat and increase performance. It looks like we have 18 total operations per second, but 0 index scans per second, which might mean we aren’t indexing and could have some query performance inefficiencies.  Going into the PostgreSQL hosts navigator, we can view things like changes in operations per second, transactions, and disk usage to ensure our system can handle current workloads and maintain consistent performance:    We can also click into a specific host to view individual host metrics like how many transactions succeeded, failed, or were rolled back and the cache hits versus disk hits, both of which impact overall performance:  Moving between Infrastructure, APM, and Log Observer Our database monitoring journey will most likely start at the service level or with the applications they back, so let’s dig into query performance and how to view its impacts on overall application performance.  From within our PostgreSQL host navigator, if we select a specific host, we can view logs or related content in APM to view services that have a dependency on the currently selected host:  We can then jump to the Database Query Performance to view and analyze query time, latency, and errors to see which specific areas are impacting response time and user experience and where we might be able to optimize our query performance:  Closing this out, we can see our Service Map and where the current database sits so that we can investigate specific errors, traces, or related logs: We moved from Infrastructure to Application Performance Monitoring, but we could have just as easily started with our Service Map and began troubleshooting database performance issues from there using Database Query Performance, database requests/errors, or traces.  Wrap Up Monitoring key metrics from the databases that power our applications is critical to the performance and reliability that our users count on. Configuring the OpenTelemetry Collector to receive PostgreSQL telemetry data and export this data to a backend observability platform is an easy process that provides invaluable visibility into the databases that back our services. If you’d like to try exporting your Postgres data to Splunk Observability Cloud, try it free for 14 days!   Resources Automatic Discovery and Instrumentation of PostgreSQL with Splunk OpenTelemetry Collector Database Monitoring: Basics & Introduction OpenTelemetry Collector Configuring Receivers ... View more

Introduction In our last post, we went over Splunk Synthetic Monitoring basics to kickstart proactive performance monitoring, improve user experience, and meet SLAs. Now let’s dig into more detail and build a Browser test using the Google Chrome Recorder.  Using the Chrome Recorder to build out Browser tests is the recommended way to capture complex and critical user flows like signup, login, and checkout. It’s simpler and more resilient than manually targeting elements using things like XPath expressions and lets you quickly get up and running with Synthetic Monitoring.  After we use the Google Chrome Recorder to record an interaction with our online boutique e-commerce website, we’ll import our recording into Splunk Synthetic Monitoring. Once imported, we’ll organize our test, view the results, and alert on failures. To follow along, you’ll need the Google Chrome Browser and access to Splunk Observability Cloud (psst! Here’s a 14 day free trial).  Building a Browser Test For our online boutique, checkout is the most critical business process, so we’d like to monitor it using Splunk Synthetic Monitoring. To do this, we’ll create a recording of the checkout flow by following the record, replay, and measure user flows example in the Chrome DevTools Docs.  With our Product Checkout recording complete, we’ll export it from the browser as JSON:  Moving over to Splunk Observability Cloud and navigating to Synthetics, we can use this recording to create our Browser test.  First, we’ll add a new Browser test: After we configure our new test by setting the necessary values, we can import our recording by selecting Import, (side note: you won’t be able to select Import until you provide a name for your test): Once the JSON file is uploaded, we can continue to edit our test, or we can try out our new test to make sure the configuration is valid by selecting Try now…: We’ll see output from our test run, but these results are ephemeral and don’t impact our overall test run metrics. It looks like our test run was successful, so let’s take a moment to celebrate how easy that was! Now on to fine-tuning. Test Organization It looks like our test is made up of one big, long interaction, which isn’t super helpful for future troubleshooting purposes:  Transactions help us break our Synthetic tests into logical steps that represent user flows. Right now, it looks like our test has one step, when in fact, we took multiple steps (browsing the catalog, adding an item to the cart, actually placing the order) when we were recording the interaction with our site. If we go back and edit our test to include transactions, we’ll be able to scope our results to each transaction and quickly identify the exact points where we encounter performance issues. Let’s see what this looks like.  First, we’ll close out of our Try now results. Then we’ll select Add synthetic transaction, which will add a new transaction section:  Let’s name our first transaction Home Page. We’ll delete this auto-populated Click step and drag our first “Go to url” step into this new transaction:  We’ve gone ahead and organized the remaining steps into transactions:  Let’s see what a test run looks like with these more discrete transactions:  The Business Transactions section of our run results is now broken down into our defined transactions. We can click on these transactions to filter filmstrip and waterfall results and also use them to identify, at a glance, when a step in our test fails (we’ll see this in a bit).  Adding Assertions and Detectors Before we call this test good, we need to add some assertions so that our test will actually succeed/fail on defined success/failure conditions. It would also be helpful to receive a notification whenever our test fails so we can resolve any issues before our customers are impacted. Let’s close out these Try now results and continue editing our test.  To create an assertion, we first add a step to the transaction we want to validate. This will auto-populate with a Click action. If we select the Click action and expand the dropdown, we can scroll down to view the available Assertions:  In our Home Page transaction, let’s assert the text “Free shipping with $75 purchase” is visible, so we know we’ve successfully loaded the HTML for our page: We could also validate the presence or absence of elements on the page by adding assertions for things like specific products. These types of assertions are more robust and help test out database connections to further ensure critical paths are up and running. After we’ve added assertion steps to each of our transactions, we can Return to test and submit our first Browser test.  Note: refreshing the page or selecting Editing Checkout Process at the top of the page won’t save any of the current changes. If you want to save progress, it’s best to submit the test and then make incremental edits along the way so you don’t lose updates.  Our test is now active and running, and if we select our test from the Overview page, we can see the results:  We don’t yet have line graph charts for the last day, 8 days, and 30 days since we just created this test, but we do have Uptime Trends, Availability, and Performance KPIs. We can select a test run from the Recent run results or a plot point from our Availability chart to view test run results.  From our run results page, we can see right away that our test failed thanks to the red banner at the top of the page:  We can also easily see which transaction failed because we should have 5 transactions, but instead, we only have 3. It looks like the assertion we set on our Add to Cart transaction failed so the other 2 transactions didn’t execute.  Rather than constantly watching test runs, let’s add a detector for these kinds of failures. We could have added a detector when we initially configured our test: Or we can add detectors from our test details page:  We’ll create a detector and name it Checkout Process Downtime. This detector will alert on Downtime that exceeds the given threshold of 10%. Every failed test run contributes to this downtime threshold, so if test run failures exceed our set threshold, we’ll get alerted. When creating a detector, we can conveniently see how frequently it will alert based on the thresholds we set so we can fine-tune them: Wrap Up That’s it! We now have a Splunk Synthetic Monitoring Browser test imported from the Google Chrome Recorder. This test will ensure our critical checkout workflow is performing as expected and alert us when it’s not so we can resolve issues before our users are impacted.  If you’re ready to build confidence around your user’s experiences, meet SLAs, and maintain a competitive edge when it comes to your application’s performance, start by building out your own Splunk Synthetic Monitoring Browser tests. Either head over to Synthetics in Splunk Observability Cloud to get started or sign up for a Splunk Observability Cloud 14 day free trial.  Resources Using the Chrome Recorder With Splunk Synthetic Monitoring Use a Browser test to test a webpage Running Synthetics Browser Tests ... View more

In our last post, we mentioned that the 3 key pieces of observability – metrics, logs, and traces – provide invaluable backend system insight so that we can detect and resolve failures fast. But how can we work towards proactively preventing system failures before they ever surface to our customers?  Frontend monitoring provides insight into what users experience when interacting with our applications. Splunk provides solutions for Real User Monitoring (RUM) to monitor actual user interactions and Synthetic Monitoring to simulate end-user traffic. While both are critical to observability and Digital Experience Monitoring (DEM), we’ll start with Splunk Synthetic Monitoring in this post. Let’s explore how Splunk Synthetic Monitoring works and what it brings to our observability practice.  Synthetics Monitoring in Action Navigating to Synthetics from Splunk Observability Cloud, we land on the Overview page: Here we can see a list of all of our existing Synthetic tests. We can filter to look for a specific type of test, sort the list by clicking on the table headings, or create new tests and global variables (variables that are shareable throughout Browser and API tests – a good place to store login credentials, for example). If we select Add new test, we can specify which type of test we want to create: Browser test, API test, or Uptime test. Browser Tests Browser tests simulate the run of a workflow or set of requests that make up the user experience and continuously collect performance data on these requests. They can be executed from many devices and from a number of locations around the world to ensure that no matter how users access applications or where they access them from, they’ll experience consistent performance. They also create a cool filmstrip with screenshots and a video replay for easy viewing of all actions executed during the session and their results in the browser. Detectors can be configured to alert on any errors or latency encountered during the test run.  Here we’ve created a simple Browser test:  This runs in the AWS-hosted Splunk public locations of Frankfurt, London, and Paris on a desktop device with a standard network connection and a 1366 x 768 viewport. Our test runs every minute and runs through one location at a time (round-robins), rather than concurrently hitting all locations. This particular test doesn’t yet have detectors configured, but if we selected Create detector, we could easily setup detectors and alerts for if/when this test fails:  Browser tests can hit a single page or be transactional with multiple steps. You can use this to evaluate real customer transactions, like logging in to your site or buying a product. In this example test, we have 7 steps that execute different possible paths a user can take when interacting with our e-commerce website. The first step includes multiple transactions. The Home transaction goes to a specified URL for our site. The following Shop transaction executes the provided JavaScript to select a random product from a list of products sold on our site:  Then the product is added to a shopping cart, an order is placed, checkout is confirmed, and the test returns to keep browsing products:  You can see that the Place Order transaction contains multiple steps. Different actions and selectors are available within each step. Additionally, steps can be imported via JSON files generated from the Google Chrome Recorder, but we’ll go into this in a separate post.  One note: you aren’t limited to running these tests on your own sites. It’s possible to run Synthetic tests against other sites, like those of competitors, to benchmark your performance against them. API Tests API tests check the transactional functionality and performance of API endpoints. They verify that endpoints are up and running and return the correct data and response codes. Alerts can also be configured on API tests based on any part of the HTTP request or response.  When we create a new API test, we configure setup steps and request steps by selecting Add requests: Setup steps include the actions required to set up the request, and the request step is the actual body of the API request. Validation steps allow you to check the response body, run JavaScript on the response, save, or extract values.  Here we have a test that hits the Spotify API: In the first request we grant server-to-server access by providing a Spotify authorization token we have specified as a global variable for our Synthetic tests. This way, when the token changes, we don’t need to edit every test that uses it, just the variable. In the validation step, the $.access_token is extracted from the Spotify response and saved to a custom variable. This variable (custom.access_token) is then used in a subsequent request to search for a specific track name:  We can extract values from the response body in the validation step or do things like assert the response body or headers contain certain values, assert the response code is what we expect, etc.  Uptime Tests Uptime tests can either be HTTP tests or Port tests. They don’t parse HTML, load images, or run JavaScript but make a request and collect metric data on response time, response code, DNS time (for HTTP tests), and time to first byte. HTTP tests hit a specified URL or endpoint, and Port tests make a TCP (Transmission Control Protocol) or UDP (User Datagram Protocol) request to the specified server port.  Test History If you open up any of the Synthetic tests from the Overview page, you’ll land on the Test History page. Here you’ll find a summary of performance trends, Key Performance Indicators (KPIs), and recent run results:  At the top of the page, we have line graph charts showing data trends for the last day, 8 days, and 30 days. The bar chart summarizes the overall results for the given time period. The Performance KPI chart is a customizable visualization with adjustable settings. You can view test run details by selecting any point in the chart or by selecting a test run from the Recent run results table below the KPI chart.  Every Browser test run generates additional charts and metrics, and the interaction between the test runner and the site being tested is represented as a waterfall chart on the test run results page. Browser test run results also include a filmstrip screenshot of site performance and a video of the site loading in real time. This lets you see how the page responds in real-time and see exactly what a user trying to load your site would see. When there’s a problem, you can select the APM link next to the related network activity to jump directly into APM and see what in your backend services may have contributed to the issue. Browser tests capture 40+ metrics, (including core web vitals), that you can use to extend your view into site performance by configuring charts, dashboards, and detectors using these custom metrics.  Wrap Up You’re now ready to set up your first Browser, API, or Uptime test to find, fix, and proactively prevent performance issues that could affect key user transactions. Don’t yet have Splunk Observability Cloud? Try it out free for 14 days!  Resources Introduction to Splunk Synthetic Monitoring in Splunk Observability Cloud Introduction to Splunk Synthetic Monitoring What Is Splunk Synthetic Monitoring Why You Need Synthetic Monitoring ... View more

Introduction We know three key players in observability are metrics, traces, and logs. Metrics help you detect problems within your system. Traces help you troubleshoot where the problems are occurring. Logs help you pinpoint root causes. These observability components, (along with others), work together to help you remediate issues quickly.  In our previous post, we discussed how Splunk Observability Cloud can help us detect and troubleshoot problems specifically in our Kubernetes environment. But how can we use our telemetry data to identify exactly what’s causing the problems in the first place? In this post, let’s dig into Splunk Log Observer Connect and see how we can diagnose and resolve issues fast.  Splunk Log Observer Connect Overview Splunk Log Observer Connect is an integration that makes it possible to query log data from your existing Splunk Platform products (Enterprise or Cloud) and use the data alongside metrics and traces all from within Splunk Observability Cloud. If you’re a Splunk Enterprise or Splunk Cloud Platform customer, you can use Log Observer Connect to view in-context logs, run queries without SPL, and jump to Related Content with one easy click to quickly detect and resolve system problems.  You can get started with Log Observer Connect by following the setup steps or working with your Support team to add a new connection for Log Observer Connect in Splunk Observability Cloud.  Using the native OpenTelemetry logging capabilities deployed as part of the Helm chart included in the Splunk Distribution of the OpenTelemetry Collector is the recommended way to get logs from Kubernetes environments into Splunk. You can also configure logging during the initial integration process of the OTel Collector by specifying Log collection and providing your Splunk HEC endpoint and access token:  Splunk Log Observer Connect in Action Interacting with logs in Splunk Observability Cloud often begins with an alert triggered by some error event like a problem with a Kubernetes cluster.  In Splunk Infrastructure Monitoring we can see in the Kubernetes Navigator, (which we toured in a previous post), that we have two such active alerts firing:  Opening them up, we can see critical alerts for memory usage. With a single click, we can Explore Further in Splunk Application Performance Monitoring by clicking on the Troubleshoot link:  This will take us to a Service Map view of our application where we can see something isn’t right:  Our paymentservice node is highlighted in red, meaning it’s the source of the root cause of our errors. If we select the red circle, we’ll see more info in the panel on the right and Infrastructure and Logs Related Content at the bottom of the screen. All of this information is specifically scoped to the selected paymentservice.  We can expand the Logs Related Content:  And then jump directly from there to Log Observer to view logs related to this service error:  With help from the logs, we can get to the bottom of what’s causing these errors. Let’s add some additional filters in the Content Control Bar to filter our logs by keywords or field values. Since we arrived via Related Content, we already have logs filtered to service.name = paymentservice. If we only wanted to see logs related to paymentservice errors, we could add another filter for severity = error:  If at any time we wanted to save a query to later validate a fix or share it with the rest of our team, we could add it to our Saved Queries. Select Save at the top right of the screen, then Save query to name and describe the saved query for later use:  Other users and/or your future self can use the Saved Query dropdown to later apply your saved query:  Moving over to the Fields panel on the right of the screen, we can view all available metadata present on entries in the Logs table. This is a great place to filter logs if you’re unsure of what fields you’re looking for. Here, we can see there’s a k8s.cluster.name field with the top values listed. In this case, we know which Kubernetes cluster we want to isolate, so we can include all logs for our specific cluster of interest:  We can then click on an individual log entry to see its details:  From the log details, we can see that the error message is “Failed payment processing through ButtercupPayments: Invalid API Token(test-20e26e90-356b-432e-a2c6-956fc03f5609).” Selecting the error message, we can filter further with a single click of Add to filter to ensure all logs are scoped to those with this error message: We’ve also added the version field as a column by selecting the kebab menu next to the field followed by Add field as column: Now we can easily scan the Logs table and identify which errors are associated with which version. At a glance, it appears that all the error logs are related to the same version number. Suspiciously, if we look for the version field in the Fields list, we can see that there is in fact only one version scoped to the current error logs: Before we jump to resolutions, we can continue to interact with the log details and move through our system. We can explore the traces related to this error and see where in our code this error is being thrown. This could help us track down any recent code changes that may have potentially caused this error. We can simply click on the trace_id in the log detail and then View trace_id to jump to the trace back in Splunk APM:  If we open up one of the span errors and go into Tag Spotlight for the version trace property, we can confirm our suspicions – only our latest release is experiencing this “Invalid API token error”:  If we had first discovered the error while investigating this trace, we could have initially gotten to Log Observer via the Related Content from either the trace:  Or the Tag Spotlight: Wrap Up We used Log Observer Connect to easily locate the cause of our errors. Thanks to the ability to move between Splunk Infrastructure Monitoring, APM, and Log Observer, we were able to confidently move forward with a fix 🎉.  If you want to connect your Splunk Enterprise or Splunk Cloud Platform logs to Splunk Observability Cloud using Splunk Log Observer Connect, again, check out the Introduction to Splunk Log Observer Connect.  New to Splunk and want to get started with Splunk Observability Cloud? Start a 14 day free trial!  ... View more

Intro In a Kubernetes environment, you can scale your application up or down with a simple command, a UI, or automatically with autoscalers. However, to scale successfully, you need to know when you’re hitting scaling limits and if/when your scaling efforts are effective. Otherwise, you might continue to inefficiently use resources or hit application performance issues unnecessarily. In this post, we’ll check out Kubernetes Horizontal Pod Autoscaling (HPA), when you might use HPA, caveats you might hit when scaling pods, and how you can use Splunk Observability Cloud to gain insight into your Kubernetes environment to ensure you’re scaling efficiently and effectively.  Kubernetes Autoscaling Autoscaling is an awesome way to increase the capacity of your Kubernetes environment to match application resource demands with minimal manual intervention. With autoscaling, scalable resources automatically increase or decrease with variable demand. This creates a more elastic, more performant, and more efficient (both in terms of application resource consumption and infrastructure costs) Kubernetes environment. Kubernetes supports both vertical and horizontal scaling. With vertical scaling (up/down), resources like memory and CPU are adjusted in place (think increasing/decreasing memory for an existing workload). Whereas with horizontal scaling (in/out), the number of replicas increases or decreases (think increasing/decreasing the number of workloads). Vertical scaling is great for right-sizing your Kubernetes workflows to ensure they have the resources they need. Horizontal scaling is great for dynamically scaling to meet unexpected bursts or busts in traffic to distribute the load.  Horizontal and vertical autoscaling can be configured at the cluster and/or pod level using Cluster Autoscaling, Vertical Pod Autoscaling, and/or Horizontal Pod Autoscaling. The Horizontal Pod Autoscaler (HPA) is the only autoscaler included by default with Kubernetes, so we’ll keep our focus on HPA for now.  Horizontal Pod Autoscaling To scale a Kubernetes workload resource like Deployments or StatefulSets based on the current demand of resources, you can manually scale workloads, or you can automatically scale workloads through autoscaling. Scaling up or down automatically to match demand reduces the need for manual intervention and ensures efficient resource use within your Kubernetes infrastructure. If load increases, horizontal scaling will respond by deploying more pods. Conversely, if load decreases, the HorizontalPodAutoscaler will instruct the workload resources to scale down, as long as the number of pods is above the configured minimum.  Horizontal Pod Autoscaling Gotchas Automatically scaling pods is a hugely beneficial feature of Kubernetes, but there are some caveats when implementing Horizontal Pod Autoscaling. Here are some things to be aware of:  Metric lag: because the HorizontalPodAutoscaler continuously checks the Metrics API for resource usage in order to inform scaling behavior, there can be a lag between monitoring usage and scaling. HPA checks metrics every 15 seconds by default. Vertical scaling conflicts: VPA and HPA shouldn’t be used together when based on the same metrics – this can lead to competing and conflicting scaling decisions.  Resource limits: if requests and limits aren’t properly configured, HPA might not be able to scale out. Fine-tuning thresholds can be tricky and requires monitoring resource limits. Resource competition: new pods spinning up can compete for resources and can also take time to initialize and stabilize. Not all applications can easily scale horizontally (single-threaded applications, those with order-dependent queues, databases, etc.). Before implementing HPA, you need to determine application compatibility.  DaemonSets: HPA doesn’t apply to DaemonSets – if you want to scale your DaemonSet, you probably should scale your node pool instead.  Dependency bottlenecks: external dependencies (such as 3rd party APIs) might not scale at the same rate or at all – you should have a plan to scale those as well. Let’s HPA Now that we know what Horizontal Pod Autoscaling is and some things to be aware of when working with HPA, let’s see it in action.  We have a PHP/Apache Kubernetes deployment under the Apache namespace that is exporting OpenTelemetry data to Splunk Observability Cloud. Our deployment creates a new StatefulSet with a single replica. Let’s jump into the Splunk Observability Cloud Kubernetes Navigator, which we explored in a previous post.  In the Navigator, if we filter down to our cluster and the namespace Apache, we can see that we currently only have one pod in our node:  The pod is receiving some significant load, and for HPA example purposes, we have deliberately limited the resources for each Apache pod. We can see spikes in CPU and memory usage that are leading to insufficient resources: The lack of required resources is throwing containers into a CrashLoopBackOff. For a minute we’ll have 1 active container:  Then suddenly, that container will crash and we’ll have 0 active containers before it attempts to restart again: Not only can we see these containers starting and stopping in real-time, but the restarts triggered an AutoDetect detector that would have notified our team of an issue: The Kubernetes Navigator helped us identify our resource issues and the impacts they’re having on our containers, but now we need to resolve these issues. Let’s now set up Horizontal Pod Autoscaling so our workload will automatically respond to this increased load and scale out by deploying more pods. First, we’ll create our HPA configuration file under our ~/workshop/k3s/hpa.yaml directory:  The HorizontalPodAutoscaler object specifies the behavior of the autoscaler. You can control resource utilization, set the min/max number of replicas, specify the direction of scaling (up/down), set target resources to scale, etc. We’ll apply the configuration by running kubectl apply -f ~/workshop/k3s/hpa.yaml.  We can see that the autoscaler was created and we can validate Horizontal Pod Autoscaling with the kubectl get hpa -n apache command. Here’s what the response looks like:  Now that HPA is deployed, our php-apache service will autoscale when either the average CPU usage goes above 50% or the average memory usage for the deployment goes above 75% with a minimum of 1 pod and max of 4 pods. In the Kubernetes Navigator nodes view, we can validate that we now have 4 pods to handle the increased load. We’ve added a filter to highlight the 4 pods in the Apache namespace:  Looking at our K8s pods tab, we can see additional pod-level metrics and again verify the number of active pods is now 4.  If we wanted to increase the number of pods to 8, we could simply update our hpa.yaml and specify 8 maxReplicas. Once deployed, we can see we now have 8 active Apache pods:  After configuring our HorizontalPodAutoscaler, we can sit back and watch our container count remain steady as our pods autoscale to handle the increased traffic.  Wrap Up  If you’re interested in automatically scaling Kubernetes workloads to match increased load with minimal manual intervention, Horizontal Pod Autoscaling might be for you. Before you get started, watch out for some of those common gotchas we mentioned. To identify pods running heavy on resource utilization and where you might benefit from setting up HPA check out the Splunk Observability Cloud Kubernetes Navigator. Don’t have Splunk Observability Cloud? We got you. Start a Splunk Observability Cloud free trial! Ready to jump into the Kubernetes Navigator? Get started integrating Kubernetes and Splunk Observability Cloud! ... View more

Intro In our previous post, we walked through integrating our Kubernetes environment with Splunk Observability Cloud using the Splunk Distribution of the OpenTelemetry Collector for Kubernetes. In this post, we’ll look at the general Splunk Distribution of the OpenTelemetry Collector and dive into the configuration for a Collector deployed in host (agent) monitoring mode. We’ll walk through the different pieces of the config so you can easily customize and extend your own configuration. We’ll also talk about common configuration problems and how you can avoid them so that you can seamlessly get up and running with your own OpenTelemetry Collector.    Walkthrough After you’ve installed the OpenTelemetry Collector for Linux or Windows, you can locate configuration files under either the /etc/otel/collector directory for Linux or the \ProgramData\Splunk\OpenTelemetry Collector\agent_config.yaml for Windows. You’ll notice several Collector configuration files live under this directory –  a gateway_config used to configure Collectors deployed in data forwarding (gateway) mode, an otlp_config_linux configuration file for exporting OpenTelemtry traces to Splunk, configuration files designed for use with AWS ECS tasks, etc. Because we’re looking at configuring our application’s instrumentation and collecting host and application metrics, we will focus on the agent_config.yaml Collector configuration file. When you open up this config, you’ll notice it’s composed of the following blocks:  Extensions Receivers Processors Exporters Service Extensions In the extensions block of the Collector config, you’ll find components that extend Collector capabilities. This section defines things like health monitoring, service discovery, data forwarding – anything not directly involved with processing telemetry data.  The Splunk Distribution of the OpenTelemetry Collector defines a few default extensions:  health_check: sets an HTTP URL that can be hit to check server availability and uptime http_forwarder: accepts HTTP requests and optionally adds headers and forwards them smart_agent: gets metrics for the host OS  zpages: serves zPages, an HTTP endpoint for live debugging different components  Receivers  Receivers are responsible for getting telemetry data into the Collector. This section of the configuration file is where data sources are configured.  In this example config file, we have several default receivers configured:  fluentforward: receives log data through Fluentd Forward protocol hostmetrics: collects metrics about the system itself jaeger: receives traces in Jaeger format otlp: receives metrics, logs, and traces through gRPC or HTTP in OTLP format prometheus: collects metrics from the collector itself – hence the /internal smartagent: legacy SignalFX Smart Agent monitors used to send metric data (soon to be deprecated in favor of native OpenTelemetry receivers) signalfx: receives metrics and logs in protobuf format zipkin: another Collector-supported trace format Processors  Processors receive telemetry data from the receivers and transform the data based on rules or settings. For example, a processor might filter, drop, rename, or recalculate telemetry data.  batch: without parameters tells the Collector to batch data before sending it to the exporters. This way, not every single piece of telemetry data is sent off to the exporter as it’s processed.  memory_limiter: limits the memory a collector can use to ensure consistent performance resourcedetection: detects system metadata from the host – region, OS, cloud provider, etc.  Exporters  This is the configuration section that defines what backends or destinations telemetry data will be sent off to.  sapm (Splunk APM exporter): exports traces in a single batch to optimize network performance signalfx: sends metrics, events, and traces to Splunk Observability Cloud  splunk_hec: sends telemetry data to a Splunk HEC endpoint to send logs and metrics to Splunk Enterprise or Splunk Cloud Platform splunk_hec/profiling: sends telemetry data for AlwaysOn Profiling to a Splunk HEC endpoint otlp: sends data through gRPC using OTLP format, especially useful when deploying in forwarding (gateway) mode debug: enables debug logging Service The service block is where the previously configured components (extensions, receivers, processors, exporters) are enabled within the pipelines.  telemetry.metrics: emit telemetry metrics about the Collector itself extensions: where all previously defined extensions are enabled for use pipelines: defined for each type of data – metrics, logs, traces Problems There are a few problems you might run into when configuring your OTel Collector. Common issues are caused by:  Incorrect indentation Receivers configured but not enabled in a pipeline  Receivers enabled in a pipeline but not configured  All receivers, exporters, and processors used in a pipeline must support the particular data type (metrics, logs, traces) Indentation is a very common problem. Collector configs are in YAML, which is indentation-sensitive. Using a YAML linter can help you verify that indentation has been maintained successfully. The good news is that the Collector fails fast – if the indentation is incorrect, the Collector will not start so you can identify and fix the problem. If you’ve set up your Collector, but data isn’t appearing in the backend, there’s a high chance receivers are being configured but subsequently aren’t enabled in a pipeline. After each pipeline component is configured, it must be enabled in a pipeline under the service block of the config.  If the specified data type in receivers, exporters, and processors aren’t supported, you’ll encounter an ErrDataTypeIsNotSupproted error. Confirm the pipeline types of the different Collector components to ensure the data types in the config are supported.  You can always ensure your Collector is up and running with the health check extension, which is on by default with the Splunk Distribution of the OpenTelemetry Collector. From your Linux host, open http://localhost:13133. If your Collector service is up and running, you’ll see a status of “Server available”.  You can also monitor all of your Collectors with Splunk Observability Cloud’s built-in dashboard. Data for this dashboard is configured in the metrics/internal section of the configuration file under the Prometheus receiver.  Wrap up  To help you through the configuration of your own OTel Collector, we walked through the config file for the Splunk Distribution of the OpenTelemetry Collector and called out potential problems you might run into with the config. If you don’t already have an OpenTelemetry Collector installed and configured, start your Splunk Observability Cloud 14 day free trial and get started with the Splunk Distribution of the OpenTelemetry Collector.  Resources Splunk OTel Collector Configuration Overview OpenTelemetry Collector Configuration Collector components Tutorial: Configure the Splunk Distribution of OpenTelemetry Collector on a Linux host ... View more