If you’ve enabled Automatic Discovery in your install of the Splunk Distribution of the OpenTelemetry Collector by following along with Part 2 of this series, then you’re already sitting back, relaxing, and watching it automatically detect and monitor services like your PostgreSQL databases. But what about more complex scenarios? What if you’re running multi-tenant databases where each tenant has its own schema? Or managing a microservice architecture with dozens of interconnected services? What about monitoring an entire application stack – databases, caches, web servers, and message queues. How do you monitor all of these things without manually configuring each component? Automatic Discovery. In this final part of this Automatic Discovery series, we’ll explore practical real-world use cases that go beyond basic setup. You’ll see how to use discovery rules, conditional configuration, and labeling to handle complex production scenarios, all while maintaining the hands-off automation that makes Automatic Discovery so powerful. Note: Automatic Discovery supports specific infrastructure services – primarily databases, caches, and select message queues. The exact list of supported services depends on which receivers are bundled in your Collector version, and not every receiver bundled with the Collector supports Automatic Discovery. For the full list of Automatic Discovery third-party application supported in Kubernetes and Linux, check out the documentation on Automatic Discovery of Apps and Services. Database Observability at Scale Databases are perfect candidates for Automatic Discovery because they follow predictable patterns that often require environment-specific configuration. If you followed the setup in Part 2, you’ve already seen basic PostgreSQL discovery in action. Now let’s explore more advanced database monitoring scenarios. The Splunk Distribution of the OpenTelemetry Collector includes bundled receiver configurations for common infrastructure services. These work automatically once you enable discovery.enabled:true in your Helm values. The examples in this section show how to customize those bundled configurations for specialized scenarios like advanced metric collection or environment-specific settings. For more standard database deployments, the bundled configurations handle discovery automatically without any additional configuration beyond providing credentials. Advanced PostgreSQL Discovery with Custom Metrics If you want to customize which PostgreSQL metrics are collected or you need to monitor specific databases based on pod labels, here’s how to extend the bundled PostgreSQL receiver configuration in your Helm values: This configuration extends the default PostgreSQL discovery to: Enable TLS verification for secure connections Collect specific database performance metrics beyond the defaults Use environment-based credentials for security Monitor all discovered PostgreSQL instances Note: We’re illustrating common PostgreSQL receiver capabilities, and the actual metric names, configuration options, and available settings may differ depending on your receiver version. Collection intervals also depend on your receiver’s capabilities and Collector version. Reducing or increasing collection intervals will impact the volume of data sent to Splunk Observability Cloud, which can impact your metric billing and cardinality. MySQL/MariaDB Discovery MySQL and MariaDB work similarly to PostgreSQL – the bundled configurations handle basic discovery automatically. Here’s how to configure credentials in your Helm values: The Collector will automatically discovery any MySQL of MariaDB pods in your cluster and start collecting metrics using these credentials. Note: don’t forget to create the mysql-monitoring secret in your cluster before deploying, just like you did for PostgreSQL in Part 2 of this series: Monitoring Multiple Services in Your Application Stack Real applications often consist of multiple interconnected services – databases, caches, message queues, web servers. Let’s say you’re running a typical web application with: PostgreSQL database for persistent data Redis cache for session storage MongoDB for document storage The application itself Here’s what a complete Helm values would look like in order to enable discovery for all of them: Deploy this configuration once, and the Collector will automatically discover and monitor every PostgreSQL, Redis, and MongoDB instance that appears in your cluster. Handling Production versus Non-Production Environments One common question that often comes up around Automatic Discovery is, “Can I use the same discovery configuration in development and production?” The short answer: yes, with minor adjustments. One of the benefits of Automatic Discovery is that the same configuration works across all environments, however, you’ll generally want different collection intervals or different credentials depending on the environment. The recommended approach is to use the same Helm values configuration across environments but create environment-specific secrets. For example, in development, your secrets might look like: And in production it might look like: To adjust collection intervals with environment-specific values, simply update the collection_interval value in each environment-specific values file: values-production.yaml: values-development.yaml: Then deploy the appropriate values file: Beyond Databases: Other Supported Services Automatic Discovery isn’t limited to databases. The Splunk Distribution of the OpenTelemetry Collector includes bundled configurations for other common infrastructure services like: Databases: PostgreSQL MySQL/MariaDB MongoDB Redis Message queues: RabbitMQ Kafka Web servers: NGINX HTTP Server Here’s a basic example of Helm values for Redis discovery: Once your Redis instance is deployed, you’ll see metrics start flowing into Splunk Observability Cloud. Next Steps Automatic Discovery reduces manual work, decreases the potential for observability gaps, and eliminates lag time between deploying infrastructure and gaining visibility into it. If you’re ready to get started with Automatic Discovery, here’s a checklist summary of the steps to ensure deployment goes smoothly: The beauty of Automatic Discovery is that once it’s enabled, every new service you deploy gets monitored automatically. Mo more extensive manual configuration. No more observability gaps. Don’t yet have Splunk Observability Cloud? Try it free for 14 days! Resources Automatic discovery of apps and services Getting started with the Splunk Distribution of the OpenTelemetry Collector Collector Components: Receivers ... View more

In Part 1 of this series, we covered what Automatic Discovery is and why it’s critical for observability at scale. Now let’s get hands-on. Enabling Automatic Discovery is pretty straightforward. The key to it all is understanding the configuration patterns and security considerations that will save you time and energy down the road. By the end of this post, you’ll have Automatic Discovery running in your Kubernetes environment and know how to avoid the most common setup pitfalls. Note: this post focuses on Kubernetes deployment using Helm, which is the recommended approach for most production environments. The examples use Helm values configuration that maps to the underlying OpenTelemetry Collector configuration. Prerequisites: What You Need Before Getting Started Before jumping into configuration, make sure you have: Splunk Observability Cloud access and a valid Ingest Access Token for the Collector to send metrics The version 0.109.0 or higher (required for the splunk.continuousDiscovery feature gate that enables Automatic Discovery in the UI) Kubernetes cluster access or a Linux environment (Automatic Discovery for third-party services is currently available for Kubernetes and Linux only) Network connectivity from your Collector to target services Note: If you’re running the Collector on Windows, Automatic Discovery of third-party services (databases, caches, message queues) is not currently supported. Instead, you’ll need to use traditional manual receiver configuration. Windows does, however, support zero-code APM instrumentation for backend applications. Step 1: Enable the Discovery Extension When deploying the Splunk Distribution of the OpenTelemetry Collector using Helm, you enable Automatic Discovery through your Helm values configuration. Here's what that basic setup looks like:  The enabled:true flag and featureGates: splunk.continuousDiscovery setting are both required. The feature gate enables the Automatic Discovery interface in Splunk Observability Cloud’s Data Management UI, where you can view all discovered services: As we discussed in Part 1 of this series, observers are how Automatic Discovery detects new services in your environment. The configuration in Step 1 includes the k8s_observer, which is automatically configured when you enable discovery in Kubernetes. The k8s_observer watches for: Pods running infrastructure services Services exposing these workloads Nodes in your cluster The auth_type: serviceAccount setting tells the observer to use the Collector’s Kubernetes service account for API access, which is the recommended approach for security and simplicity. For Linux hosts (non-Kubernetes): use the host_observer instead, which monitors processes and endpoints on the local machine. Step 2: Configure Credentials for Discovered Services Most infrastructure services require authentication. You configure credentials through the properties.receivers section in your Helm values file, referencing environment variables for security: This configuration applies to all discovered PostgreSQL instances. The Collector will automatically use these credentials when connecting to any PostgreSQL database it discovers in your cluster. Step 3: Inject Credentials Using Kubernetes Secrets The environment variables referenced in Step 2 are populated from Kubernetes secrets. Add the extraEnvs section to your Helm values to bind secret data to environment variables: Before deploying, create the Kubernetes secret in the same namespace as your Collector: This approach keeps credentials secure and separate from your Helm values files, which can be safely stored in version control. Note: if you’re monitoring multiple database types, like PostgreSQL and MySQL, add these additional databases to the receivers section of your Helm values file, add the additional environment variables to the extraEnvs section, and create additional secrets for each: Step 4: Deploy  With these individual configuration components in place, we can now complete deployment. Add/update the Splunk OpenTelemetry Helm repository: Create the namespace: Deploy using your Helm values file: Step 5: Verify Automatic Discovery To verify that Automatic Discovery is working, you can verify pods are running and watch the Collector logs for Automatic Discovery events:  Output should look something similar to this: You can also check the Data Management UI in Splunk Observability Cloud, as mentioned in Part 1, to validate Discovered Services. Best Practices Automatic Discovery is powerful, but to get the most out of it, here are some best practices we recommend based on what our customers have done. Start Small and Expand Gradually Start with one or two service types, like PostgreSQL, to understand the discovery patterns in your environment. Once you’re comfortable with how services are detected, credentialed, and monitored, you can scale up to cover additional infrastructure. This incremental approach helps prevent unexpected issues from propagating across the cluster. Use dedicated monitoring users Avoid using application or admin accounts for monitoring. Instead, create service-specific read-only users for databases and other infrastructure services. This minimizes risk and ensures your observability setup has just enough access to gather metrics without exposing sensitive data. Secure secrets thoughtfully Never hardcode credentials in your Helm values or configuration files. Use Kubernetes secrets, and consider integrating external secret managers. This not only improves security but also allows easier credential rotation without redeploying your Collector. Adopt consistent labeling conventions While not required, consistent Kubernetes labels on your workloads make discovered services easier to organize and track in Splunk Observability Cloud. Labels also help when setting up dashboards, alerts, or filtering metrics for specific clusters, namespaces, or environments. Monitor discovery health proactively Set up alerts or dashboards to track the health of your discovery extension. Watch for failed discovery attempts, authentication issues, network connectivity problems or unexpected disappearance of services. Catching issues early prevents blind spots in your monitoring. Document your configuration thoroughly Maintain clear records of what services are being monitored, where credentials come from, and how receivers are configured. Include team ownership, escalation paths, and any customizations. This documentation is invaluable when onboarding new team members or troubleshooting incidents. This information can be provided in runbooks with your incident response tool (such as Splunk On-Call), making it easier for you to solve the inevitable issues that arise without having to spend time searching for this data. Test changes in non-production environments first As always, validate configuration updates in development or staging environments. Automatic Discovery is dynamic, and testing first ensures you won’t unintentionally disrupt monitoring in production. What’s Next? With Automatic Discovery configured, you now have automatic monitoring for your infrastructure services. Next, in Part 3 of this series, we’ll explore practical applications like automatically discovering and monitoring more of your applications, including databases, web services, and messaging systems. We’ll also get into advanced patterns for complex multi-services applications. Missed the beginning of this series? Check out Automatic Discovery Part 1: What is Automatic Discovery in Splunk Observability Cloud and Why it Matters. Ready to get started with Automatic Discovery? Set it up with Splunk Observability Cloud’s free for 14 day trial. Resources Automatic discovery of apps and services Getting started with the Splunk Distribution of the OpenTelemetry Collector Exploring the OpenTelemetry Collector’s Kubernetes annotation-based discovery ... View more

If you’ve ever deployed a new database cluster, spun up a caching layer, or added a load balancer, you know it can involve a lot of manual configuration, YAML wrangling, and crossing your fingers that you didn’t miss anything critical. Automatic Discovery in the Splunk Distribution of the OpenTelemetry Collector helps eliminate some of this toil so you can automatically and easily begin monitoring your infrastructure and applications in Splunk Observability Cloud the moment they come online. Traditional Observability Setup Challenges Setting up observability for modern infrastructure can be a lift. You deploy a new database instance, and suddenly you’re writing Collector configurations, setting up service discovery, and manually defining which metrics to collect. Scale that across dozens of services, multiple environments, constant deployments, and you’ve got a recipe for potential observability gaps. With traditional approaches, you need to: Manually configure monitoring for each new service Update Collector configs every time infrastructure changes Maintain separate configurations for different environments Hope that your teammates remember to add monitoring when they deploy a new service And the result of all this? Potential blind spots in your observability coverage and time lost to configuration rather than innovation and problem solving. Automatic Discovery to the Rescue Automatic Discovery thankfully flips this model on its head. Instead of manually configuring monitoring for every service, you enable Automatic Discovery once, and the Splunk Distribution of the OpenTelemetry Collector automatically detects, configures, and starts monitoring new services as they appear. Automatic Discovery detects and collects signal data from third-party services such as databases and web servers by automatically generating configuration snippets that you can modify and incorporate into your existing configuration. Think of it as a smart assistant that constantly scans your infrastructure, recognizes common services, and automatically sets up monitoring based on what it finds. How Automatic Discovery Works With discovery mode enabled, the Collector performs intelligent detection through a multi-step process. Observer Extensions Scan Your Environment  Automatic Discovery uses observer extensions – like the k8s_observer in Kubernetes or the host_observer in Linux – to continuously watch for infrastructure endpoints. These observers poll at short intervals and report discovered endpoints (pods, services, processes) to the Collector typically within seconds of them becoming available. Preflight Testing with Bundled Configurations  The Splunk Distribution of the OpenTelemetry Collector includes bundled receiver configurations for common infrastructure services like PostgreSQL, Redis, Kafka, NGINX, etc. During a "preflight" discovery phase, the Collector tests these bundled configurations against the endpoints discovered by the observers to see which ones successfully connect and collect data. This means that newly deployed services typically begin reporting metrics within moments of starting up. Automatic Integration of Successful Discoveries  When a receiver configuration successfully discovers a service (for example, detecting a database on a specific port), the Collector automatically incorporates that receiver into its active configuration. Metrics start flowing to Splunk Observability Cloud immediately – no Collector restarts or manual configuration updates required.  Discovery State and Persistence Discovery state is maintained in memory by the Collector during runtime. When a service is discovered and monitoring begins, the Collector tracks that endpoint as long as it remains active. If the Collector restarts, it performs a fresh discovery scan of the current environment state at startup, re-evaluating all known observer sources like nodes, pods, or network interfaces. This means it will rediscover any services that are still running but won’t retain information about services that stopped while the Collector was down. For ephemeral services, like containers or scaled-down pods, the Collector stops attempting to collect metrics from those endpoints. In Splunk Observability Cloud, stopped services typically drop from the active discovery list after the next few polling cycles (usually within minutes). With everything up and running, you can verify what’s been discovered by checking the Collector logs for discovery events or by viewing the new services and metrics that appear automatically in Splunk Observability Cloud’s Infrastructure Navigator. Real-World Impact If you were to deploy a new cache cluster with traditional manual configuration, before you could get any metrics you would need to: Update the Collector configuration with receiver settings Specify connection details Restart the Collector Verify metrics are flowing If you happen to be running multiple cache instances across different namespaces, you’d need to repeat this process for each one. With Automatic Discovery enabled, you deploy your new cache cluster, and the Collector immediately detects the new pods and automatically begins collecting metrics. Within seconds, you see cache-specific metrics – no manual Collector configuration, no YAML editing, no hoping someone remembers to add monitoring. This same pattern applies for all your supported infrastructure services. Deploy it, and it’s automatically monitored. The Benefits of Automatic Discovery Eliminates Observability gaps With traditional manual configuration, it’s easy to miss new services or forget to update monitoring when services change. Reduces Time to Visibility Instead of waiting for someone to manually configure monitoring, new services become observable the moment they start running. This is critical for fast-moving teams practicing continuous deployment. Scales Effortlessly Whether you’re managing 10 services or 1,000, Automatic Discovery scales with your infrastructure. Define your rules once, and they apply consistently across all environments. Reduces Configuration Drift Manual configurations tend to drift across environments and teams. Automatic Discovery ensures consistent monitoring configuration based on your defined rules, reducing environment-specific quirks. Enables Self-Service Observability Development teams can deploy new services with proper monitoring automatically – no observability team intervention required. Where Automatic Discovery Fits in Your Observability Strategy Automatic discovery isn’t meant to replace thoughtful observability design – it’s meant to eliminate the tedious manual work that prevents you from focusing on what matters. Here’s how it fits into the bigger picture: Discovery layer: Automatic Discovery handles the detection and basic configuration of monitoring for standard services and patterns. Customization layer: for services that need specialized monitoring or custom metrics, you still have full control over Collector configuration. Alerting layer: once Automatic Discovery sets up basic monitoring, you can layer on custom alerting rules, SLOs, and business-specific metrics using Splunk Observability Cloud’s full feature set. Think of Automatic Discovery as providing the observability foundation that lets you focus on higher-value observability practices like building meaningful dashboards, setting up intelligent alerting, and creating SLIs that matter to your business. Getting Started: What’s Next? Automatic Discovery shines brightest in dynamic environments where services are constantly being deployed, scaled, and updated. If you’re running Kubernetes workloads, containerized applications, or cloud-native infrastructure, Automatic Discovery can drastically simplify your observability operations. The setup process is straightforward, but there are important considerations around security, configuration management, and best practices that can make the difference between a smooth rollout and a frustrating experience. In Part 2 of this series, we’ll walk through the step-by-step process of enabling Automatic Discovery, including how to securely handle credentials, avoid common pitfalls, and configure discovery rules that work for specific infrastructure patterns. Ready to stop manually configuring observability for every new service? You can explore Automatic Discovery by setting it up with Splunk Observability Cloud – try it free for 14 days and see how it simplifies your setup. Resources Automatic discovery of apps and services Getting started with the Splunk Distribution of the OpenTelemetry Collector Exploring the OpenTelemetry Collector’s Kubernetes annotation-based discovery ... View more

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, but something is off – conversion rates are dropping, and Customer Support is getting complaints about slow checkout. Your monitoring shows that all servers are green, response times look normal, and there are no errors in the logs. But revenue is down 15% from where it was last year at this point. What’s going on? With Business Transactions in Splunk Observability Cloud and Business iQ in Splunk AppDynamics, we can understand what’s happening technically in our critical user flows and get immediate insight into business impact. Understanding when to use each of these tools can mean the difference between fixing a random technical issue and solving a business-critical problem. Let’s dig in and explore how these tools can work together to bridge the gap between code and commerce.   Splunk APM Business Transactions: Your User Journey Monitor Business Transactions in Splunk Application Performance Monitoring previously known as Business Workflows, bring proven transaction monitoring to modern architectures. Business Transactions group related traces that represent discrete user flows – like User Registration, Add to Cart, or Complete Purchase. They turn your distributed microservice traces into meaningful business processes you can monitor and optimize. Business Transactions build real business context around technical pieces of your application so you can quickly gain insight and investigate real issues. It can be difficult to diagnose the impact when getting alerted on hundreds of failed SQL queries, but when you can quickly see that the Checkout process is taking users 7 seconds longer than usual, you can easily prioritize and jump in to investigate – this is how Business Transactions help you break down the many technical pieces of an application to create logical, meaningful context. In this example, the first transaction in our Business Transaction list is customer checkout, and the send is a health check for our Ad Service. Business Transactions are perfect for: Monitoring end-to-end user journey performance across hybrid environments Tracking transaction-level SLAs and success rates Understanding how deployments affect critical customer flows Setting up alerts on business-critical transactions instead of for individual services (reducing noise vs. individual service alerts) Real-World Example Your mobile app team reports that user registrations are down 20%. Business Transactions reveal that the Complete User Registration flow now takes 12 seconds instead of the previous 3 seconds. Drilling down, you discover a new Email Verification Service causing the delays. Fix deployed, and registrations recover within hours.   Splunk AppDynamics Business iQ: Your Revenue Impact Calculator Business iQ is a comprehensive observability analytics tool that helps you translate application performance data into immediate business impact analysis. Implementing Business iQ means you can analyze business data like revenue, conversion rates, and customer drop-offs alongside IT data like application performance metrics, traces, and logs. It doesn’t just inform you that checkout is slow, it tells you: exactly how much money the slowness is costing you, which customer segments are most affected, and where to focus your optimization for maximum ROI. Business iQ is perfect for: Quantifying the business impact of performance issues Prioritizing optimization efforts by revenue potential Demonstrating ROI of performance improvements Creating executive dashboards that speak business language Monitoring business-critical KPIs like sales, conversions, and customer satisfaction Release validation with business impact analysis Real-World Example Your Payment Processing Service is experiencing intermittent timeouts. Traditional monitoring shows a 2% error rate – not terrible. But Business iQ reveals that those 2% failed transactions represent $250,000 in lost revenue per day, concentrated in your highest-value enterprise customers. Suddenly, this becomes your top priority.   Which Tool When? Scenario Your Go-To Tool Why? “Our checkout flow is slower than usual” Business Transactions Tracks transaction performance “How much revenue are we losing from slow checkouts?” Business iQ Calculates business impact “Which user journeys need optimization?” Business Transactions Shows transaction-level metrics “Which performance issue cost us the most money?” Business iQ Prioritizes by revenue impact “Did our deployment affect critical flows?” Business Transactions Compares transaction performance “What’s the ROI of fixing this performance issue?” Business iQ Quantifies financial benefit   Better Together These two tools aren’t competitors, they’re complementary layers of the same observability stack. Here’s an example of how they might work together: Business iQ alert: revenue is down 10% over the last hour Business Transactions: Checkout transaction error rate spiked to 15% APM deep dive: Payment Service is showing timeouts to an external gateway Business iQ validation: a fix was deployed, and revenue recovery was confirmed   Wrap Up Business Transactions turn your technical monitoring into contextual user journey insights. They help you understand how your code performs from the user’s perspective and ensure your most critical flows stay healthy. Business iQ turns your performance insights into business intelligence. It maps application performance to business performance, connecting the dots on how performance drives or disrupts revenue, customer engagement, or the critical metrics your business cares about. Working together, these tools can bridge the gap between your engineering team’s performance metrics and your executive team’s goals. Ready to connect code performance to business impact? Check out the Splunk Documentation or join the conversation in the Splunk Community.   Resources Configure Business Transaction Rules Monitor Cloud-Native & Hybrid Apps and Business Transactions with Observability Cloud APM AppDynamics Analytics documentation Transforming Data into Business Impact with AppDynamics Business iQ E-Book ... View more

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running slower than usual, and it’s majorly impacting user experience. Where do you start your investigation? Do you dive into the application Service Map in Splunk Application Performance Monitoring (APM) to trace request lows? Or do you check the Service Analyzer tree view in Splunk IT Service Intelligence (ITSI) to see the hierarchical business service health to understand business impact and prioritize response?  This is kind of a trick question because the reality is that it depends on which team you work on and what problems you're trying to solve. Thankfully, Splunk gives you the complete visibility you need, and when you’re equipped with the understanding the who and when that goes along with each of these tools, you’ll have the power to solve root causes rather than just fixing symptoms. So let’s learn more about each.   APM Service Maps: Your Application GPS If you're on an engineering, SRE, or DevOps team, chances are you'll spend a lot of time with APM Service Maps. Service Maps provide a real-time visualization of how your microservices are connected and how they’re performing in relation to each other and their dependencies. Every technical component of your application – every service, dependency, and service interaction – becomes a node connected by a line on the map. You can see your entire application architecture from a birds-eye view. From this view, you also get directed troubleshooting information like root causes for errors and quick access to services to get information specific to that service in the Service Map. Service Maps are perfect for: Troubleshooting microservice communication issues Understanding distributed application architecture Finding performance bottlenecks in service-to-service calls Monitoring real-time application health Real-World Example Your Checkout service is throwing 500s. You open the APM Service Map and immediately see that the Payment service is healthy, but there’s a large red node representing the Inventory service. Drilling in, you discover that the Inventory service database is timing out. What looked like a Payment service issue is actually an Inventory service problem affecting the entire checkout flow. Note: You may have heard "Flow Maps" referenced in other products like Splunk AppDynamics. These serve the similar purpose of visualizing how transactions or data flow across systems in a more technical way, but the implementation and focus differ.    ITSI Service Analyzer Tree View: Your Business Impact Compass The ITSI Service Analyzer tree view takes observability up a level – from technical components to business services – and aggregates data across tools for first-line responders and Ops teams. It organizes your technology around business outcomes and can be used as a first line of support to diagnose and triage incidents from a high level. Instead of showing individual microservices, they show logical business services and their relationships like Web Front End to Product Catalog and how underlying technology components support them. This topographical tree view allows you to quickly investigate issues. For example, if we click into our yellow On-Prem Database node, we can dig deeper and get an expanded view of specific KPIs that are causing an issue, view and investigate relevant episodes for the service, and move into Deep Dive to further troubleshoot. Service Analyzer tree views are perfect for:   Understanding business impact of technical issues Prioritizing incident response based on business impact Communicating technical status to business stakeholders Monitoring overall service health across the organization Real-World Example Your CEO asks about the impact of this afternoon’s infrastructure issue on customer-facing services. The Service Analyzer tree view shows that while 15 individual servers had problems, only the internally used “Reporting Dashboard” business service was affected. Customer-facing services remained green and functional. Crisis communication avoided.   Which Tool When?  Scenario Your Go-To Tool Why? “Users can’t complete checkout transactions” APM Service Maps Traces microservice interactions “The customer portal is down, what’s the business impact?” ITSI Service Analyzer tree view Shows business service health “We need to understand our service dependencies” ITSI Service Analyzer tree view Hierarchical dependency view “Microservice A is throwing errors to Microservice B” APM Service Maps Inter-service communication “Which infrastructure issues affect our revenue systems?” ITSI Service Analyzer tree view Business-aligned service health   Complementary Strategy While each of these visualizations is independently helpful and can provide strong observability value, they can also complement each other to give deep cross-team, cross-functional visibility into complex environments from different levels. Here’s how they might work together in a real issue investigation: Alert fires: Customer Portal is slow First-line support team checks ITSI Service Analyzer tree view: is this impacting business-critical services? Engineering, SRE, DevOps teams jump into APM Service Map: which microservices are having issues? Once the fix is in, revisit ITSI Service Analyzer tree view: are all business services recovered and healthy?   Wrap Up APM Service Maps outline the technical conversation between your application’s services. The ITSI Service Analyzer tree view translates technical complexity into business impact. While they don't have to be used in conjunction, they can be used together so that all operational and technical teams can have a complete picture that will speed up troubleshooting, help you quickly assess impact, and ultimately benefit your entire organization. Ready to build out your complete picture? Check out the Splunk Documentation or join the conversation in the Splunk Community.   Resources View dependencies in the service map in Splunk APM Investigate the root cause of an error with the Splunk APM service map Use the Service Analyzer tree view in ITSI ... View more

It’s 3 pm on a Friday, and your application is running slower than it should be. Support tickets are coming in and you want to solve the issue fast – for your users and for your weekend. With so many capabilities for monitoring, troubleshooting, and performance analysis, which tool will give you the insight you need right now? Thankfully, Splunk Observability Cloud quickly gives you the complete visibility you need to optimize performance. AlwaysOn Profiling helps identify bottlenecks before they become incidents or support tickets. And now, Call Graphs – the battle-tested feature from Splunk AppDynamics – helps troubleshoot in-progress performance issues for faster resolution. But which one do you use and when? Let’s talk it out.   AlwaysOn Profiling: Your Early Warning System AlwaysOn Profiling continuously samples your application’s call stacks, taking snapshots every 10 seconds (by default) for CPU and periodically for memory. These get aggregated into flame graphs where the width of each box shows how much time your code spends in each method: AlwaysOn capabilities are perfect for: Finding memory leaks before they cause outages Identifying inefficient algorithms or code paths Catching performance regressions after deployments Pre-deployment performance testing Real-World Example Your application has been gradually getting slower. No single transaction is terrible, but everything just seems a bit sluggish. You open profiling and immediately see 40% of CPU time is spent in a JSON serialization method that someone “optimized” last month. Mystery solved in minutes, not hours. Performance Impact AlwaysOn Profiling is not on by default and requires explicitly enabled. This intentional setup ensures you’re aware of the additional monitoring. While AlwaysOn Profiling is designed for continuous operation with low overhead (typically single-digit percentage impact), it’s still sampling your application’s call stacks every 10 seconds. Test out the impacts in your own development environment to understand the specific impacts before launching it in Production.   Call Graphs: Your Incident Detective Call Graphs show you the exact method-by-method execution path of individual requests. Unlike profiling’s aggregate view, this is transaction-specific detail.  You see the exact sequence of method calls, how long each took, and which external calls were made. Call Graph capabilities are perfect for: Investigating specific slow transactions Quick incident triage and team routing Root cause analysis of intermittent issues Validating that fixes actually worked Quickly identifying which teams are involved in incidents without needing to read any code (a win for Operations teams) Real-World Example A customer reports checkout took 45 seconds. You find their trace, pull up the Call Graph, and there it is: a third-party payment API is taking 43 seconds to respond. You see the exact method that made the call. Ticket routed to the fixing team, case closed. Performance Impact Like AlwaysOn Profiling, Call Graphs require explicit configuration in your APM setup. Call Graphs capture detailed method-level timing for each traced request, which can add overhead if applied to 100% of transactions. The default 5% sampling rate strikes a balance between diagnostic capability and performance impact and typically captures enough data for troubleshooting while keeping overhead minimal. Once configured, you can tune method execution thresholds (default filters out methods under 10ms) and exclude certain packages to reduce overhead in your specific environment and focus on the bottlenecks that are meaningful to you.   Which Tool When? Scenario Your Go-To Tool Why? “Our app has been getting slower over the past week” AlwaysOn Profiling Shows trends over time “This customer can’t complete checkout” Call Graphs Traces exact transaction “We need to optimize before Black Friday” AlwaysOn Profiling Proactively identifies bottlenecks “The CEO just hit a 30-second page load” Call Graphs Immediate transaction detail “Did our deployment slow things down?” AlwaysOn Profiling Compares across deployments   Better Together Even though these two tools might sound similar, they really work better in conjunction. When investigating an issue, here’s a typical flow that takes advantage of the capabilities provided by both AlwaysOn Profiling and Call Graphs: Alert fires: response time degraded by 50% Check profiling: is this affecting all requests or specific operations? Find patterns: profiling shows increased CPU in image processing methods Drill down with Call graphs: see exactly which image processing operations are slow Root cause: new high-resolution images uploaded by the Marketing team exceeded processing limits   Wrap Up While they might sound similar on paper, AlwaysOn Profiling prevents problems by continuously monitoring performance trends and catching regressions early – it’s a proactive tool. Call graphs solve problems by showing exactly what happened in slow transactions – it’s a reactive tool. You definitely need both to prevent fires before they start and to quickly put them out after they do.   Ready to level up your performance game? Try Splunk Observability Cloud (including AlwaysOn Profiling and Call Graphs) free for 14 days.   Resources Splunk Observability Cloud Documentation Call Graphs Optimizing application, service and memory usage with AlwaysOn Profiling for Splunk APM - Splunk Lantern Memory Profiling for Java Applications with AlwaysOn Profiling | Splunk ... View more

In our observability journey so far, we've built comprehensive instrumentation for our Worms in Space application. In our first post, Manual Instrumentation with Splunk Observability Cloud: The What and Why, we explored the fundamentals of manual instrumentation. Our second post, Manual Instrumentation with Splunk Observability Cloud: How to Instrument Backend Applications, walked through backend instrumentation with OpenTelemetry. In our third post, Manual Instrumentation with Splunk Observability Cloud: How to Instrument Frontend Applications, we added frontend observability with Splunk RUM. Now, let's enhance our observability architecture by introducing the OpenTelemetry Collector – a vendor-agnostic way to receive, process, and export telemetry data for better security, reliability, and flexibility. Why Use the OpenTelemetry Collector? Before we dig into the implementation, let’s understand why the OpenTelemetry Collector is a best practice when it comes to production-ready observability. Current Architecture: Direct to Cloud With our current application architecture, telemetry data from our frontend and backend is pushed directly to Splunk Observability Cloud. This approach works, but it has its limitations around: Security: backend services need access tokens in their configurations Reliability: each service handles its own buffering and retries Flexibility: updating or changing out observability backends requires updating every service Processing: there’s no central place for data enrichment or filtering Improved Architecture: OpenTelemetry Collector The OpenTelemetry Collector solves these problems and acts as a central gateway that: Centralizes authentication: the Collector is the only thing that needs observability access tokens Handles retries: retry logic, batching, and queuing is all built-in Enables flexibility: observability backends can easily be switched out and updated in one configuration file Adds processing: filtering sensitive data, enriching telemetry data, and sampling can all take place in the Collector Improves performance: application services no longer need to handle exporting telemetry data   Implementing the OpenTelemetry Collector Now that we know the whys behind using the OpenTelemetry Collector, let’s walk through implementing it in our application step-by-step. The complete code can be found in our Manual Instrumentation GitHub repository. Step 1: Add OpenTelemetry Collector Service and Configuration Git commit: ebc5145 First, we create the Collector configuration file, otel-collector-config.yaml: To get a deeper explanation of the configuration, check out our previous blog post on OpenTelemetry Configuration & Common Problems. Some of the key configuration elements to note are: Receivers: OTLP protocol support for both gRPC (port 4317) and HTTP (port 4318) Exporters: OTLP/Splunk:  exports distributed traces to Splunk SignalFX: processes Splunk’s metric data and has different ingestion requirements than the OTLP/Splunk exporter Debug: provides local visibility during development and troubleshooting Processors: Batch: groups telemetry data into larger payloads for efficient network transmission Resource: enriches telemetry with additional metadata like deployment environment Health check extension: provides a health check endpoint for Docker container health monitoring CORS configuration: critical for browser-based RUM to send traces The multi-exporter approach in our config ensures optimal data flow for different telemetry types. Since we’re using Docker in our application, we also need to add the collector to Docker Compose. You could also deploy the collector as a sidecar in K8s deployment or run it through another service orchestrator. Here’s the section we’ll add to our docker-compose.yml: The Splunk distribution of the OpenTelemetry Collector includes optimizations and additional features specifically for Splunk Observability Cloud. You can choose to use the upstream distribution, but as a Splunk customer, you’ll get support for the Splunk Distribution from our team. Step 2: Configure the Backend to Send Traces to OpenTelemetry Collector Git commit: 9da3a37d Now we need to update our Elixir backend configuration to include the OpenTelemetry Collector in our config/runtime.exs file: We’ve removed the direct Splunk configuration that set the realm and access token, and instead, pointed the OTLP endpoint to the local Collector service. We can then remove authentication headers, since authentication and routing is now handled in the Collector. Step 3: Configure Frontend RUM to Send Traces to OpenTelemetry Collector Git commit: 929ea749 Finally, we can update the frontend RUM configuration to send traces to the OpenTelemetry Collector. To do this, we’ve updated our assets/js/rum.ts to the following: Note: our beaconEndpoint here points to a locally running Collector in our development environment. To make this code production ready, we could deploy a public Collector endpoint, and then replace this beaconEndpoint value with that new endpoint. Another option would be to deploy the Collector as a sidecar, as we mentioned above. Optional Step 4 for Brownie points: Add Environment Variable Examples for OpenTelemetry Collector Setup Git commit: 58826200 This last commit simply adds environment variable examples to our .env.example file to guide users through configuring Splunk Observability Cloud credentials and service settings: Centralizing this configuration through environment variables keeps them out of the code and makes deployments easier. Wrap Up In just four commits, we've successfully integrated the OpenTelemetry Collector into our full-stack application. This architectural improvement provides a production-ready telemetry pipeline that scales with your needs while maintaining the comprehensive observability we've built throughout this series. By centralizing telemetry collection and processing, we’ve simplified our application observability, improved security, and provided the flexibility needed for evolving and adapting to any future observability requirements. Best of all, our space worms can continue to schedule their spacewalks with confidence, knowing that every aspect of the application is observable through a robust, scalable telemetry pipeline. Ready to implement the OpenTelemetry Collector in your own application? Start with the Splunk OpenTelemetry Collector documentation and a Splunk Observability Cloud 14-day free trial to see your telemetry data in action. Resources OpenTelemetry Collector Documentation Splunk OpenTelemetry Collector OpenTelemetry Collector Configuration Splunk Observability Cloud Documentation ... View more

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many locally hosted LLMs. But how do you understand the performance of those models and whether they’re impacted by other services? In this post, we’ll look at how Splunk Observability Cloud can help you gain this insight into LLM-based applications to troubleshoot and find root causes just as quickly as we can with our other applications and services.  Troubleshooting Latency in an LLM-based Application We’ll start by logging in Splunk Observability Cloud, and navigating to Splunk Application Performance Monitoring (APM). We’ll make sure we’re in the correct environment for our application, and set the time range to the last 1 hour:  At a high level, our application uses multiple LLMs to answer questions from end users. One of these LLMs is gpt-4o-mini from OpenAI, and the other is an open-source LLM named MistralAI, which we’re self-hosting on Nvidia GPU hardware. The application was built using the Langchain framework, and utilizes a Chroma vector database for Retrieval Augmented Generation (RAG). It’s instrumented with the to automatically capture our telemetry data so we have full visibility into its performance.  Looking at the screenshot above, it looks like we have a critical alert firing in one of our main services, the auto-prompter service. Let’s take a closer look by selecting that “1 Critical” label next to the service name to open up the alert details:   In the alert, we can see that the service is experiencing higher latency than normal. Let’s click on the Troubleshoot hyperlink in the Explore Further section on the lower right side of the screen to see what’s going on for this service in APM:  Over in APM, we can see that our auto-prompter service has started experiencing some significant latency spikes. Let’s select the Traces tab and open up a long running trace to see where the latency is originating:  In this trace, we can see that spans have been captured for each major step used by our application to respond to the question.    We first see a GET request to our auto-prompter service to build a prompt, which kicks off the process of sending the request into a couple different LLMs to get a response. There’s a call to our Chroma vector database to look for similar documents, and finally, a data-scrubber service, which uses an LLM to remove any personal identifiable information (PII) from the response.  From this trace waterfall view, we can see that the bulk of the execution time for this trace is spent calling the LLM.  We can select the specific span where the auto-prompter service invokes the LLM to get a closer look:  In the Span Details, we can see from the duration that it took the LLM over 21 seconds to respond to the auto-prompter service. Let’s easily move over to Splunk Infrastructure Monitoring by selecting the three dots next to the auto-prompter service name in this pane and clicking on the top link to go to the OpenAI Instance Navigator. This will quickly allow us to have a look at the underlying infrastructure: With this dashboard, we can see details related to the OpenAI-compatible models utilized by our auto-prompter service. This includes the total number of input and output tokens used, as well as the tokens used broken down by service and model.   Nothing unusual appears to be happening here, and the token usage by the auto-prompter service is low.  Let’s explore a bit more by switching from the overview to the table view, so we can see the other services using the OpenAI framework: From this view, we can see all the services utilizing OpenAI-compatible LLMs, along with the model they’re using.   It looks like another service named otel-genai-zero-code is using the same Mistral LLM, and with 1.5 million output tokens in the past hour, this service is probably consuming the bulk of the available capacity of this LLM’s underlying GPU infrastructure.  We’ll validate this by navigating to Infrastructure, selecting AI Frameworks, and then opening up the Nvidia GPU navigator:  From the Nvidia GPU navigator, we can clearly see that the GPU resources used by this LLM are nearing 100%.   We can resolve this “noisy neighbor” issue by either working with the team that manages this service to reduce its usage of the LLM, allocating more GPU resources, or we can pull out our application to its own dedicated GPU hardware. It also would be a good idea to create a detector and alert to proactively notify us if another application ends up using too many tokens in the future before this puts too much strain on our GPU.  Wrap Up  Not only does Splunk Observability Cloud provide full visibility into our standard applications and services, it also offers full insight into applications and services that utilize LLMs and Nvidia GPU hardware. We were able to use Splunk Observability Cloud to quickly find the root cause of performance issues within our LLM-based application and quickly resolve the issue to get our application back to an optimally performing state. Want to implement full observability in your LLM-based applications and services? Try Splunk Observability Cloud free for 14 days! Resources Top Features of Splunk Observability Cloud for Engineers Splunk Observability Hub ... View more

Although it might seem daunting, as we’ve seen in this series, manual instrumentation can be straightforward and a powerful tool in a complete observability practice. In this series, we dig into manual instrumentation to see how we can use it to easily build out observability from scratch. In our first post, Manual Instrumentation with Splunk Observability Cloud: The What and Why, we learned about what manual instrumentation is and why you need it. Our second post went through the process of manually instrumenting the backend of a real, un-instrumented full-stack application, commit by commit. In this post, we will look at our un-instrumented frontend and go step-by-step through the process of manually instrumenting it.  Example Application Overview Recap GitHub repository: https://github.com/splunk/evangelism-public/tree/main/manual-instrumentation To review, our example application is called Worms in Space. It consists of a React frontend with an Elixir backend and provides a full GraphQL API. Worms in Space allows customers (who happen to be worms) to schedule spacewalks in… space.  Space worms can use the UI or GraphQL API to: View available time slots Schedule space walks during predefined or custom time slots View all scheduled space walks Cancel space walks Here’s what the frontend UI looks like:  And here you can interact with the GraphQL API via GraphiQL:  If you'd like to follow along on GitHub commit-by-commit, check out the GitHub repository, otherwise, you can read on to follow each commit here. Why Splunk RUM for Frontend Instrumentation? Splunk RUM (Real User Monitoring) within Splunk Observability Cloud is designed to monitor the performance and user experience of web and mobile applications from the perspective of actual end-users. It collects data directly from the user's browser or mobile device, and provides insights into how real users interact with and experience your application for complete visibility. When instrumenting a frontend application, it’s important to work with a solution that provides: Support for multiple browsers and mobile applications Complete instrumentation of full user sessions without degrading performance or user experience Out-of-the-box, real-time metrics, dashboards, and charts around performance, user interactions, user sessions, and page loads AI-driven analytics to identify performance improvements across every page, resource, component, and third-party dependency Splunk RUM gives us all these things and more – plus it can be implemented in just a few commits. Let’s dig into the implementation.  Frontend Instrumentation with Splunk RUM Step 1: Add Splunk RUM Dependency Git commit: 9d23d770 For full documentation on the Splunk RUM installation process, check out the installation docs. We’re going to install Splunk RUM via NPM, so the first thing we need to do within our frontend application is add the Splunk RUM dependency to our package.json file: @splunk/otel-web is the Splunk Distribution of OpenTelemetry for JavaScript and abstracts away many of the underlying OpenTelemetry and Splunk-specific details.    Step 2: Configure Splunk RUM Git commit: e0caea8a Next, since our application uses Typescript, we’ll configure Splunk RUM in a assets/js/rum.ts file: This configuration sets the: realm: Your Splunk Observability Cloud realm (us0, us1, eu0, etc.). You can find your realm in your Splunk Observability Cloud organization settings. The SDK automatically constructs the correct ingest endpoint from the realm. rumAccessToken: Your RUM access token. This is different from APM tokens and has limited, frontend-safe permissions. You can generate a RUM access token by navigating to Settings → Access Tokens in Splunk Observability Cloud. applicationName: Your application name as it will appear in RUM dashboards. Choose something descriptive that distinguishes this frontend from other services. deploymentEnvironment: Deployment environment (development, staging, production). This enables filtering in RUM views. Security note: RUM tokens are designed to be exposed in frontend code. They only have permission to send RUM data and cannot access other Splunk Observability Cloud resources.  Step 3: Initialize Splunk RUM in React Application Entry Point Git commit: d65649fe We initialize RUM in our application entry point in our assets/js/app.tsx before React starts. You’ll want to do this as early as possible in the application source: The installation documentation states to import or require the splunk-instrumentation.js file, or in our case, our assets/js/app.tsx file, before other files to ensure that the instrumentation runs before the application code. With this single initialization, Splunk RUM will now automatically instrument: Performance Metrics Network Activity User Interactions JavaScript Errors For a complete list of data collected by the RUM Browser agent, check out the RUM Browser data model docs.  Step 4: Add Custom Instrumentation for Business Logic Git commit: ffaec857 While automatic instrumentation covers browser metrics, we add in our manual instrumentation with custom spans to track business-critical operations that are unique to our application. In this example, this critical operation is actually scheduling a spacewalk: This is just a snippet from our AstronautTimeSlots.tsx file. For the complete implementation, including custom spans for delete operations, and delete operations, see the full source code on GitHub. These custom spans appear in RUM traces alongside automatic instrumentation, providing complete visibility into user workflows. See custom instrumentation documentation for more patterns.  Let’s now see what this manual instrumentation looks like in Splunk Observability Cloud.  What You'll See in Splunk Observability Cloud With this instrumentation in place in just three commits, we immediately gain visibility into: Key performance metrics like: user sessions, page views, JavaScript errors, and core web vitals  Browser and device analytics: device type breakdown and performance comparison Detailed session breakdowns showing individual user journeys with timelines, waterfall views, errors, and replays End-to-end trace views showing frontend and backend correlation Out-of-the-box and customizable dashboards, and alerts Now that our application is instrumented, we can go into Splunk Observability Cloud, navigate to Splunk RUM to see that our worms-in-space-frontend application is now sending RUM data: Here we can quickly see those key performance networks like page views, JavaScript errors, and web vitals.  If we click into our worms-in-space-frontend from this view we can see a more detailed, filterable view of page views, page loads, and interactions: And then selecting the specific interaction, we can go into the waterfall view to see the detailed trace breakdown of user interactions and network requests: In this waterfall breakdown, we can see custom spans that we manually instrumented like our delete_scheduled_walk. We can select this span to explore a more detailed view with the complete span tags that we manually added in our AstronautTimeSlots.tsx file:  Manual instrumentation success via Splunk RUM 🎉. Wrap Up In just three commits, we've transformed our un-instrumented React frontend into a fully observable service with Splunk RUM. The combination of automatic instrumentation provided out-of-the-box and the manual instrumentation we added for our specific business logic provides complete visibility into our frontend application performance. This level of observability transforms mysterious production issues into clearly diagnosed problems with obvious solutions so space worms can seamlessly schedule and reschedule their spacewalks at any time. Join us in our next post to follow along, step-by-step, as we update our Worms in Space application to use the OpenTelemetry Collector.   Ready to manually instrument your own frontend? Start with Splunk Real User Monitoring, and use Splunk Observability Cloud's 14-day free trial to easily visualize your telemetry data in one unified platform. Resources Manual Instrumentation with Splunk Observability Cloud: The What and Why Manual Instrumentation with Splunk Observability Cloud: How to Instrument Backend Applications Splunk Observability Cloud Overview ... View more

Manual instrumentation is an invaluable observability tool. In this series, we dig into manual instrumentation to see how we can use it to easily build out observability from scratch. In our first post, Manual Instrumentation with Splunk Observability Cloud: The What and Why, we learned about what manual instrumentation is and why you need it. In this post, we'll look at a real, un-instrumented full-stack application and go step-by-step through the process of manually instrumenting its backend, commit by commit. Preface: SDK-based Automatic Instrumentation Before we jump in, we want to call out that while we will be using manual instrumentation to create custom telemetry data, we’ll also be using OpenTelemetry’s SDK-based automatic instrumentation to hook into our web framework, HTTP requests, and database so we don’t have to manually create spans for framework operations.  A true zero-code example of automatic instrumentation would require no code changes, but since zero-code auto-instrumentation doesn’t exist (yet) for our application’s Elixir backend, we’ll use SDK-based automatic instrumentation for framework operations and full manual instrumentation for our business logic instrumentation.  Example Application Overview  GitHub repository: https://github.com/splunk/evangelism-public/tree/main/manual-instrumentation Our un-instrumented example application is called Worms in Space. It consists of a React frontend with an Elixir backend and provides a full GraphQL API. Worms in Space allows customers (who happen to be worms) to schedule spacewalks in… space.  Space worms can use the UI or GraphQL API to: View available time slots Schedule space walks during predefined or custom time slots View all scheduled space walks Cancel space walks Here’s what the frontend UI looks like:  And here you can interact with the GraphQL API via GraphiQL:  If you'd like to follow along on GitHub commit-by-commit, check out the GitHub repository, otherwise, you can read on to follow each commit here. Backend Instrumentation Implementation Again, our example application is an Elixir backend with a GraphQL API for scheduling spacewalks. Even if you aren’t using Elixir or GraphQL, you can still use the following general steps as a guide to instrument your own backend application: Add the necessary OpenTelemetry dependencies Configure OpenTelemetry Initialize instrumentation Add custom telemetry data Let's walk through each of these steps commit-by-commit to instrument our Worms in Space backend.  Step 1: Add OpenTelemetry Dependencies Git commit: 5f926245 The first thing we need to do within our backend application is add the OpenTelemetry dependencies to our dependency file, in our case our mix.exs file: The list of dependencies can be found in the OpenTelemetry Language APIs & SDKs doc under the Elixir/Erlang dependencies section. Since we’re using the core OpenTelemetry SDK, we’ll need to include OpenTelemetry, the OpenTelemetry API, and the OpenTelemetry exporter. Our Elixir application is using Phoenix for HTTP requests, so to auto-instrument our controllers, plugs, and requests, we’ll need to include the OpenTelemetry dependency for Phoenix. Phoenix uses Cowboy as its default HTTP server, so we’ll also include the OpenTelemetry Cowboy dependency so we can low-level HTTP requests. Our application uses Ecto for database interactions, so we’ll include the OpenTelemetry Ecto dependency, and then  GraphQL operations in Elixir are resolved via Absinthe, so we’ll need the OpenTelemetry Absinthe dependency, as well. Finally, the OTLP exporter is required for sending telemetry data to Splunk Observability Cloud.  Step 2: Configure OpenTelemetry SDK and OTLP Exporter Git commit: e45ee707 Next, we configure the OpenTelemetry SDK in our config/runtime.exs file: This configuration establishes our service identity, sets up the batch processor for efficient trace export, and configures the OTLP exporter to send traces directly to Splunk Observability Cloud. Note: in this current configuration, we are directly sending out telemetry data to our Splunk Observability Cloud backend. In most cases, you would want to set up an OpenTelemetry Collector to collect, process, and export telemetry data to your observability backend. In a subsequent post, we’ll look at how we can update our current configuration to use the OpenTelemetry Collector for increased resilience, flexibility, and processing capabilities. Step 3: Initialize Instrumentation Components in Application Git commit: f7113fec We next initialize the automatic instrumentations in our application.ex startup code: Notes on this setup: Cowboy Setup initializes the foundational HTTP server instrumentation Phoenix Adapter setup ensures correct integration with Phoenix's HTTP server Setup order matters. Initialize instrumentation libraries before your application starts accepting requests Include logging to help verify configuration at startup This setup initializes the dependencies referenced in the first commit in Step 1 and provides immediate visibility into HTTP requests, database queries, and GraphQL operations without any code changes. Every HTTP endpoint, database query, and resolver will automatically generate traces. Step 4: Add Custom Spans for Business Logic Git commit: 9278ef62 While SDK-based automatic instrumentation covers the framework layers, we need to add custom spans to capture our application’s unique business logic. We do this by setting attributes on specific spans within our create function heads in our resolvers.ex file so that every time a spacewalk is created, custom telemetry data with specified attributes is exported to Splunk Observability Cloud: Note: I’ve collapsed this method for brevity, but the full method and file can be viewed on GitHub in the manual-instrumentation/worms_in_space/lib/worms_in_space_web/api/resolvers/resolvers.ex file. These custom spans provide deep visibility into our spacewalk scheduling business logic, including success/failure rates, scheduling types, and performance metrics for critical operations. Some of the most common attributes to include in manual instrumentation are things like customer level or user ID and can be configured based on your unique business use-cases. Environment Configuration Finally, we set our environment variables for deployment flexibility: This configuration allows seamless deployment across development, staging, and production environments while maintaining complete trace context and correlation. What You'll See in Splunk Observability Cloud With this instrumentation in place in just four commits, we immediately gain visibility into: Complete request flows from HTTP request to database query with timing and status GraphQL operation performance with query names and execution time   Database query analysis including SQL statements and execution time Custom business metrics like the ones we added around spacewalks Error tracking with full stack traces and contextual attributes Going into Splunk Observability Cloud, we can visit the service overview of our Worms in Space backend in Splunk Application Performance Monitoring. Here we can see valuable service metrics, Service Map, Tag Spotlight, errors, and traces:  If we open the traces from this view, we can see long running traces: We can open one of those long running traces, and then we can select a span to view the custom attributes that we manually instrumented earlier:  Manual instrumentation success 🎉. Wrap Up In just four commits, we've transformed our un-instrumented Elixir backend into a fully observable service. The combination of automatic instrumentation for framework operations and manual instrumentation via custom spans for business logic provides complete visibility into application performance and behavior. When issues arise at 2 AM, we’ll now know exactly which spacewalk scheduling operation failed, how long database queries are taking, and whether the problem is in our business logic or our infrastructure. This level of observability transforms mysterious production issues into clearly diagnosed problems with obvious solutions so space worms can seamlessly schedule their spacewalks at any time. Join us in our next post to follow along, step-by-step, as we manually instrument the React-powered frontend of our Worms in Space application.  Ready to add context to your own backend apps? Start with OpenTelemetry's documentation on Language APIs and SDKs, and use Splunk Observability Cloud's 14-day free trial to easily visualize your telemetry data in one unified backend platform. Resources Manual Instrumentation with Splunk Observability Cloud: The What and Why Splunk Observability Cloud Overview OpenTelemetry Language APIs & SDKs         ... View more

If you've ever worked with distributed systems, you’ve likely felt the pain of a frontend throwing errors, while your backend is processing requests slower than usual, and somewhere between your resolvers and database queries something unknown is going wrong. But what are the specific issues, where exactly are they occurring, and why did they happen in the first place? This is exactly the kind of insight observability provides, and the benefits of having such visibility into systems are endless. Faster insight into problems means proactive prevention of disruptions and performance issues. Visibility into the root cause of issues leads to quick pinpointing of the exact lines of code that need to be fixed, which means the impact to our users is minimal or non-existent, and our customers stay happy.   Observability platforms make it easy to gain visibility into application ecosystems through unified, intuitive UIs, and automatic instrumentation. Collecting and exporting the data required for this insight is often automatic through tools like automatic discovery and zero-code instrumentation. But even when automatic instrumentation is available, there are times when only you know what really matters for your application’s success. Adding in manual or custom instrumentation can provide the full picture when it comes to specific application logic or insight related to business-specific data. Manual instrumentation can be an invaluable tool in your observability toolkit, and in this series, we're going to dig into it to see how we can use it to easily build observability from scratch. We’ll look at a real application and take it from zero observability to fully instrumented, commit by commit. In this first post, we’ll learn about what manual instrumentation is and why you need it. Then, in subsequent posts, we'll look at a real, un-instrumented application, and go step-by-step through the process of manually instrumenting it.   What Is Manual Instrumentation?  Automatic instrumentation seems almost magical. Install an agent or library, restart your service, and poof -- metrics, logs, and traces start flowing into your observability platform. But what about the parts of your code that don’t get picked up automatically? The times where your services aren’t written in automatic instrumentation-supported languages? Or the times when you need more context than what’s provided by auto-instrumentation? That’s where manual instrumentation comes in. Manual instrumentation is the process of explicitly adding observability code to applications to capture custom telemetry data. Observability platforms often provide auto-discovery, zero-code implementations – including the use of which was recently donated to the community by Splunk – that fully instrument systems. But not all languages and services are supported when it comes to auto-instrumentation. Even in cases where auto-instrumentation is possible, we sometimes need more detail. Custom instrumentation around business-specific telemetry data and specific code or custom attributes provides invaluable business context and captures the nuances for our unique applications. In these cases, manual instrumentation adds a ton of value.   Unlike with automatic instrumentation, with manual instrumentation, you take control and decide exactly what and how to instrument. This means that even if your observability platform doesn’t automatically provide visibility into your code base, and your application starts throwing errors at 2 AM, you’ll still be notified about the failures, and you'll know exactly why and where errors are occurring.  Why Manually Instrument? In cases where automatic-instrumentation for a language or service isn’t available or where out-of-the-box telemetry data isn’t capturing the full application picture, manual instrumentation is a necessity. Manual instrumentation offers full-stack observability, business logic insight, and custom metrics and attributes. Let’s look at each of these benefits. Full-Stack Visibility  Manual instrumentation helps complete the full application picture for complete visibility into performance and issues. With automatic instrumentation you can see things like degraded endpoint performance, but it won't always tell you which part of your custom sorting algorithm is causing bottlenecks. With manual instrumentation, you can add instrumentation around the specific code to identify exactly where time is being spent. Is the bottleneck in the data transformation? The third-party API call? Now you'll know. Manual instrumentation lets you add these detailed breadcrumbs throughout your code, capturing variable states, decision outcomes, and execution paths. It's like leaving yourself detailed notes about what your code was doing at each step. Business Logic Insight Your application isn't just a collection of HTTP endpoints and database calls. It's a carefully orchestrated system of business rules, calculations, and decision points. Manual instrumentation lets you trace through complex workflows like order processing, fraud detection algorithms, or multi-step approval processes. When something goes wrong with a customer's order at 2 AM, you want to know exactly which business rule failed and why. Manual instrumentation allows you to capture this business-specific insight. Custom Metrics and Attributes    Every application has its own set of unique and important metrics. Maybe it's the number of items in a shopping cart, the complexity score of a generated report, or the cache hit rate for your custom caching layer. Manual instrumentation lets you capture these application-specific metrics and add custom attributes that provide context, like customer tier, feature flags, or regional data.  Wrap up For cases where automatic instrumentation isn’t possible or doesn’t provide all the features you need, manual instrumentation gets you critical visibility to keep your applications up and running. When automatic instrumentation is available, it gets you to broad coverage quickly, but adding in manual instrumentation gets you deeper insights. Manual instrumentation helps you go beyond what’s possible with automatic instrumentation, so you have the power to see exactly what's happening, when it's happening, and why it's happening – turning mysterious production issues into clearly diagnosed problems with obvious solutions. Join us in our next post to see how you can easily go from zero observability to full visibility with manual instrumentation. Ready to take control of your observability today? Start with OpenTelemetry documentation and examples to begin instrumenting your application. If you need a platform where you can visualize all your telemetry data in one place, Splunk Observability Cloud's 14-day free trial is a great way to begin your observability journey. Resources Instrument back-end applications to send spans to Splunk APM Instrumentation methods for Splunk Observability Cloud Automatic discovery of apps and services OpenTelemetry Language APIs & SDKs   ... View more

If you’ve read our previous post on self-service observability, you already know what it is and why it matters. Self-service observability empowers teams to own their observability practice, eliminating cross-team dependencies and bottlenecks, and instead optimizes delivery and speeds up incident resolution. It also empowers your engineers to customize observability for their application’s unique needs. In this post, we’ll walk through how to implement self-service observability using Splunk Observability Cloud.  Start with Team Structure and Access Control To prevent teams from fumbling through an incident while sifting through an explosion of dashboards and charts, team structure and organization within an observability platform are critical. A successful self-service observability practice is organized and starts with team structure.  In Splunk Observability Cloud, Teams are a way to organize users into functional groups and efficiently connect them to the dashboards, charts, detectors, and alerts that matter to them. To create a Team:  Select Settings in the left navigation menu Go to Team Management  Select Create team, enter a name and description, and then Add members Save the Team by selecting Create team Team members will have access to a dedicated landing page that contains relevant information tailored to their needs. A landing page brings together team-specific: Dashboard groups Detectors triggered by team-linked alerts Favorited and mirrored dashboards (centralized dashboards that are editable across teams but customizable per team without affecting other mirrors) This setup helps everyone stay aligned and focused, whether during incidents or proactive monitoring.  For each team, team managers can be assigned to configure the proper roles and user management, including Admin, Power, Usage, and Read-only. This way, people only have access to the things they need.  Access controls can be set using Splunk Observability Cloud Enterprise Edition’s built-in RBAC by: Navigating to Settings and selecting Access Tokens Creating team-specific access tokens with the appropriate permissions Setting token usage limits to manage the maximum number of usage metrics This granular token provisioning protects your environment, enables usage tracking per team, and helps to proactively alert on usage thresholds so your teams can build and use the resources they need without unknowingly impacting cost.  Standardized Templates & Observability as Code Templating and automating the creation of observability resources increases the likelihood of adoption while also facilitating standardization through easily shareable resources. There are different ways to build templates, such as using the Splunk Observability Cloud API or the Splunk Terraform Provider. As long as your resources are in source control, there will be checks and balances, rollback capabilities, and higher developer velocity (because teams can focus on writing feature code rather than creating observability resources).  Here’s an example of how to use the Splunk Terraform provider to create a dashboard and chart in Splunk Observability Cloud:  Teams can also be specified from within the Terraform definitions so that dashboards show up within the specified Team’s landing page.  If you’d like to learn more about Observability as Code, check out our blog posts: Observability as Code and Let’s Talk Terraform.  Implement OpenTelemetry To enable self-service observability, you need to standardize the collection, processing, and export of telemetry data. OpenTelemetry provides this standardization through semantic conventions, including: Tagging conventions: service.name, deployment.environment, team Metric and span naming standards: http.server.duration Attribute requirements: cloud.provider, region, k8s.cluster.name When telemetry data is standardized, teams using observability resources can confidently find and understand their data, dashboards, and alerts. These resources are then reusable and familiar to users across teams, making it possible for incident responders to quickly jump in and provide assistance in emergencies. Using semantic conventions also helps your observability tooling aggregate across different cloud providers, frameworks, and programming languages to give a unified view of your business. Measure, Iterate, Expand Achieving Self-service observability is a journey. It is an incremental process that requires measuring adoption, identifying gaps, and iteration. Tracking adoption metrics, usage metrics, and cost can help you fine-tune your practice and deliver successful and complete visibility into your systems. We recommend you track some key metrics, below, to enable you to verify that teams are using observability, to understand and control costs, and to help justify your investment in observability. Adoption metrics to track:  Percentage of teams with at least one active detector Number of dashboards or detectors created per team Time from alert trigger to acknowledgement Support ticket volumes (this should decrease as your observability practice grows) Usage metrics to track:  Metric cardinality Time series volume Whether or not specific metrics are being used Cost optimizations: Filter unused or overly granular metrics Set rules for aggregation or dropping metrics Route data to reduce ingestion and storage costs Avoid Common Pitfalls Finally, when implementing a self-service observability practice, it’s essential to avoid common problem areas that can lead to chaos, increased cost, and operational silos. Below are some key pitfalls to watch out for, along with strategies to avoid them.  Don’t skip documentation Every observability resource should have a clear description, contact information for the owning team, and links to runbooks where applicable. It might sound extreme, but a runbook should accompany every detector. Here’s a Terraform example of a detector with proper documentation:  Don’t ignore costs With each team managing its observability, there is the potential for a telemetry data explosion or a rapid increase in the amount of telemetry data being exported to a backend observability platform. This explosion of data not only makes it difficult to process and analyze, but it also increases costs.  Within Splunk Observability Cloud, you can always get detailed insight to monitor and manage subscription usage and billing to ensure you stay within your defined limits:  In the Enterprise Edition of Splunk Observability Cloud, Metrics Pipeline Management (MPM) centralizes a solution for managing metrics and helps you easily identify metric usage in-product. You can then adjust for metric explosion in-product or adjust the metric in code, and even adjust storage settings to lower costs and improve monitoring performance.  Don’t create silos Giving every team control over their observability practice can quickly dissolve into chaos if standards aren’t in alignment. Avoid silos and promote a unified strategy by:  Using shared naming conventions Providing shared templates Promoting best practices through enablement (office hours, onboarding sessions, communication channels, internal demos, lunch-and-learns) Always encourage cross-team collaboration and create opportunities for shared learning to ensure your self-service observability practice grows and thrives.  Wrap Up With an organized team structure, consistent telemetry powered by OpenTelemetry, and observability as code, your teams will not only be able to move faster, but they will also be empowered with the insights they need to respond to issues with greater efficiency and confidence.  Ready to take the first step in your self-service observability journey? Start building your self-service observability practice today with Splunk Observability Cloud’s 14-day free trial.  Resources Self-Service Observability: How to Scale Observability Adoption Through Self-Service Building a Self-Service Observability Practice Let’s Talk Terraform Introduction to the Splunk Terraform Provider Observability as Code Enable self-service observability ... View more

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. Specifically, this series goes into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start at the beginning of this series, check out: Identifying Unknown Unknowns. In this sixth post, we’ll explore how the Splunk AI Assistant can be used to generate scripts that serve as a launchpad for automation code using the Splunk Observability Cloud API.  Generating Scripts with the Splunk AI Assistant As an engineer or administrator of Splunk Observability Cloud, I may want to automate tasks related to observability resources using the Splunk AI Assistant and the Splunk Observability Cloud API.  The AI Assistant is pretty flexible in terms of the types of questions you prompt it with, as long as the questions are related to the code integrated with Splunk Observability Cloud and the data within it. We can ask the AI Assistant to generate code that interacts with the Splunk Observability Cloud API. For example, I can ask the AI Assistant to write a Python script that adds my user to a team in Splunk Observability Cloud – all that I will need is my user ID, and the team ID I want to add my user to.  To grab my user ID, I’ll navigate to my profile in Splunk Observability Cloud and copy and save my user ID:  And then I’ll navigate to the team and grab its ID:  Note: you’ll notice my user, Caitlin Halla, is not currently a member of this team. Using my user ID and this team ID, we can then prompt the AI Assistant to write a Python script that adds me, user GQSnQ4TAwAg, to team Fuqe6XzA4AA using the Splunk Observability Cloud API. The response we get from this prompt includes the code we’ll need for our Python script:  We can then copy and paste this code into our code editor and replace the placeholders and required values, like our access token (with administrative access), with real values:  After running our script, if we navigate back to the team in Splunk Observability Cloud, we can see that my user was successfully added to this team:  Wrap up To summarize what we explored in this post, we learned how the Splunk AI Assistant can help us generate a Python script to automate specific tasks, like adding a Splunk Observability Cloud user to a specific team. This type of interaction with the Splunk AI Assistant highlights the ability to use the Assistant for Observability as Code. The AI Assistant can provide you with code that interacts with the Splunk Observability Cloud API and draft SignalFlow programs that you can then insert into charts. It can even generate the charts via Python scripts.  With the Splunk AI Assistant, observability possibilities are continually expanding. In this series, we’ve discussed some of the use cases currently available within the AI Assistant, but these will only evolve and get better with time. As features expand, we look forward to seeing you in a future AI Assistant series. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Previous post in the Splunk Observability Cloud’s AI Assistant Series: Onboarding New Hires or New Users of Splunk Observability Cloud AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Observability as Code YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the fifth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. In this series, we go into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start at the beginning of this series, check out our first post: Identifying Unknown Unknowns. In this fifth post, we’ll explore how the Splunk AI Assistant can help quickly onboard new hires and/or help those who might be new to Splunk Observability Cloud.  Onboarding New Hires Deep contextual, interpreted, and real-time data isn’t just helpful to experienced engineers - as we saw in our last post, Explaining Metrics and Providing Feedback. Imagine you’re an engineer who has just joined a new company. You’re trying to understand the services and environment your team owns and how everything works together. You can use the AI Assistant to get an understanding of the environment and the data that’s flowing between services. For example, you might want to know what kind of data is being sent to the Payment service in your team’s Online Boutique environment:  From this prompt, the AI Assistant uses log data to identify what’s happening in the Payment service and returns the data types being sent and processed. In the response, we can see details on how the Payment service is processing payment transactions, how HTTP requests are made to the Payment service, order confirmation integration with an email service, and retry mechanisms for payment processing:  Finally, we have a nice summary of what the Payment service does:  This kind of information is super powerful for anyone who is new to the organization and trying to quickly get up to speed on how service architecture and how they work together. Documentation is helpful during onboarding, but the AI Assistant provides real-time, up-to-date information quickly and interactively.    Wrap up To summarize the use case explored in this post, we’ve used the Splunk AI Assistant to onboard new hires and build familiarity with Splunk Observability Cloud. In our next post, we’ll learn how to leverage the AI Assistant for Observability as Code by generating scripts that can interact with the Splunk Observability Cloud API. Scripts like these can be expanded to create dashboards, charts, detectors, and more. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Previous post in the Splunk Observability Cloud’s AI Assistant Series: Explaining metrics and providing feedback AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Onboarding New Hires and New Users to Splunk Observability Cloud  YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. In this series, we go into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start with how to access the Splunk AI Assistant and identify unknown unknowns within a service environment, check out the first post in this series: Identifying Unknown Unknowns. In our second post, Analyzing and Troubleshooting in Real-Time, you can see how to use the AI Assistant to analyze trace errors and database query performance and gain insights into detectors and alerts. Our third post digs into how to leverage the Splunk AI Assistant to ensure compliance and optimize cost requirements. In this fourth post, we’ll explore how the Splunk AI Assistant can help explain unfamiliar or custom metrics and how it can provide real-time, contextual feedback to help analyze performance, identify optimizations, and troubleshoot faster for a reduced Mean Time to Resolution (MTTR).  Splunk AI Assistant Explains Metrics Within a company’s engineering organization, different teams may implement their own custom metrics specific to the services they own and manage. When troubleshooting an incident that spans services and teams, navigating these custom metrics and their meanings can slow down investigation and MTTR. Not only that, but engineers often utilize third-party tools that can be instrumented with custom metrics that don’t conform to internal standards or conventions. Interpreting these metrics can have a similar negative impact on troubleshooting time and MTTR. Let’s look at an example.  Our engineering team utilizes the data structure server, Redis, but I haven’t spent a lot of time working with Redis, so I’m unfamiliar with all of the metrics associated with our Redis instance. I’m on-call and dealing with an active incident, and I’m seeing a lot of Redis metrics pop up around cache hit rate percentage. I’m not sure what this means, but I can use the Splunk AI Assistant to gain more context and troubleshoot the issue.  From within Splunk Observability Cloud, I’ll navigate to Infrastructure and then select Datastores. I’ll scroll down to Redis and select Redis instances to get a detailed overview of my Redis instances:      From within the Redis instance view, I can open the AI Assistant and ask detailed questions about specific instance metrics, like this cache hit rate percentage. I can also ask it for contextual data to help me determine if the metric data is good or bad.  In my prompt, I’m going to specify the instance I’m investigating because, as you’ll remember from our previous post, Analyzing and Troubleshooting in Real-Time, the more context you provide to the AI Assistant, the more complete responses will be:  I’ve asked the AI Assistant to explain what the cache hit rate percentage metric is and how I can interpret it. The response returns analysis for this specific metric, showing that over the last four hours, the hit rate was 67%.  The interpretation of the contextual data in the response suggests that this is a moderate hit rate, indicating there is room for optimization. More details on suggestions for further optimization include reviewing the cache configuration, analyzing miss patterns, and application logic:  The response also provides important relevant Infrastructure attributes along with relevant dashboards and charts, complete with hyperlinks for further insight:  Wrap up To summarize the use case explored in this post, we’ve utilized the Splunk AI Assistant to explain unfamiliar metrics and provide real-time, contextual feedback, enabling us to troubleshoot quickly and reduce MTTR. In our next post, we’ll use the AI Assistant to streamline the onboarding process for new hires or new users of Splunk Observability Cloud to get everyone up to speed fast. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Previous post in the Splunk Observability Cloud’s AI Assistant Series: Auditing Compliance and Cost AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Explaining Metrics and Providing Feedback YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to use the Splunk AI Assistant by exploring practical, real-world, real-time examples. Specifically, this series goes into the specific Splunk AI Assistant use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics and providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code If you’d like to start with how to access the Splunk AI Assistant and identify unknown unknowns within a service environment, check out the first post in this series: Identifying Unknown Unknowns. In our second post, Analyzing and Troubleshooting in Real-Time, you can see how to use the AI Assistant to analyze trace errors and database query performance and gain insights into detectors and alerts. In this third post, we’ll learn how to leverage the Splunk AI Assistant to ensure we remain in compliance with our organization’s specific requirements. We’ll also see how the AI Assistant can analyze our infrastructure and identify areas that may be underutilized, which in turn will help us reduce costs. Compliance Organizations often implement code-level tagging requirements for compliance, security, cost, and operations. For example, it’s a requirement that all of the services in our Online Boutique environment set a tag value for the attribute tenant.level to easily distinguish between service tiers. This tenant.level attribute has three tag values – gold, bronze, and silver – as seen here in Tag Spotlight within Splunk Application Performance Monitoring:  We can use the AI Assistant to audit the data in our environment and ensure we are in compliance with this specific organizational requirement. We’ll ask the AI Assistant if any services in the Online Boutique environment aren’t using this required tenant.level tag. In the response, we can see that our Ad service is missing the required tenant.level tag:  Note: In the AI Assistant pane, if you have “Use current page filters” toggled on and you try to run this query from a specific page, like the paymentservice detailed view, you might not see the full results you expect. Querying from this page would set the context to just that of the paymentservice. If we want to interrogate our entire environment and all services instead, we would need to move up a level and execute the query from our services view that lists and sets the context as all services in our environment.  If we go to our Ad service in Splunk APM, we can verify that the data from it is in fact missing the required tenant.level tag: Cost Control Similar to what we saw in our last post about reducing alert noise and ecosystem hygiene, we can use the AI Assistant to analyze our infrastructure and help us identify areas that might not be efficiently utilized. This can help us capacity plan and reduce costs when allocating infrastructure.  We’ll have the AI Assistant analyze our Kubernetes infrastructure by prompting it to tell us whether it’s underutilized and identify nodes in Splunk Infrastructure Monitoring with all-time high CPU usage below 90% in our Online Boutique environment. The AI Assistant’s analysis identifies a node in our environment with a CPU utilization below 90%. The highest CPU utilization for this node is 73.3%, which is generally a great place to sit in terms of utilization, as it leaves room for CPU spikes. However, the average CPU utilization is generally lower than the maximum CPU utilization, and if this node is provisioned on a large instance type, like an EC2 m5.4xlarge, it’s likely over provisioned.  The response from the AI Assistant indicates that our node is operating efficiently. It also provides suggestions for potential optimizations, important attributes about the node, and links to Kubernetes charts and dashboards for continued monitoring:  Wrap up To summarize the use cases explored in this post, we’ve used the Splunk AI Assistant to audit compliance and cost by identifying expensive or underutilized infrastructure and discovering service configurations that may be out of compliance. In our next post, we’ll use the AI Assistant to explain specific metrics, contextualize data, and provide feedback. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Splunk Observability Cloud’s AI Assistant Series: Analyzing and Troubleshooting in Real-Time  AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Compliance and Cost Control YouTube video Troubleshooting Microservices with Splunk Observability Cloud and the AI Assistant for Observability YouTube video Splunk Observability Hub ... View more

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at how to use the Splunk AI Assistant by digging into some practical, real-world, real-time examples. This series explores the specific use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining metrics or providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code In our first post in this series, we saw how to access the Splunk AI Assistant and then explored how we could identify unknown unknowns within our service environment. The AI Assistant identified that our latest release could be causing high error rates and high latency. Before jumping into this second post, check out that first post: Identifying Unknown Unknowns.  This second post picks up where we left off and jumps into using the Splunk AI Assistant to: Perform deploy and release checks Investigate database query performance Interrogate active detectors and alerts Deploy and Release Checks We can use the AI Assistant to compare new code releases to previous versions. Let’s see this in action by asking the AI Assistant to compare the current latency of our Payment service to the prior version. We’ll ask it to compare Payment service performance for the current and previous versions in our Online Boutique environment and identify changes in behavior or error patterns for the latest release: The AI Assistant returns a comparison that shows the latest version of our service, version 350.10, has a much higher latency than the previous version, version 350.9:  Scrolling down to the error rate analysis, we can see version 350.10 has an error rate of over 30,000 errors, while version 350.9 has 0 errors. The analysis also shows that the common error for the latest version is “Invalid request with HTTP status code 401”: The AI Assistant also highlights observations and recommendations for further investigation and resolution of the issue:  The Observations section shows that the latency increase and error rate surge are significant and indicate “potential issues with the latest release.” The Recommendations include ideas for further investigation, analysis of code changes between versions, and rolling back to version 350.9 to maintain service stability. Analyze Trace Errors While investigating real-time issues, looking at specific trace data is helpful to get a deeper understanding of the root cause. We can use the AI Assistant to analyze trace errors for this troubleshooting in real-time use case. Here, we ask the AI Assistant to analyze trace errors specifically for our Payment service in our Online Boutique environment over the last 15 minutes: It’s important to note that being more specific with your AI Assistant queries – for example, specifying services, environments, time windows, and data types will help the assistant return more specific and useful responses.  In the response for our current prompt, we can see a list of trace errors for the Payment service with dynamic links to each of the traces: We can then ask for a deeper analysis of traces by prompting the AI Assistant with a specific trace ID:    The AI Assistant identifies the root cause of the errors as a 401 status code or unauthorized access to one of our third-party APIs: the ButtercupPayments service. The response includes the error message and error tags for the invalid request, and also an analysis of trace performance:  Scrolling down in the response, we can see the Performance Issues section that provides helpful information about the duration of the spans within our specific trace. The duration for this trace was relatively short, which hints at a pretty immediate rejection of the request. This section also highlights the service interaction, which explains how the data is requested. In this case, our Payment service is trying to request the Buttercup Payments API and failing due to this authorization issue: Looking at the recommendations, the AI Assistant suggests reviewing the authentication mechanisms for the Buttercup Payments API to ensure the credentials or tokens are valid, along with a couple of other recommendations:  Selecting the hyperlink at the end of this response will take us to the trace in Splunk APM so we can look at the data in detail and explore further: Database Query Performance Not only can we ask the AI Assistant to analyze traces and metrics related to our services, but we can also ask it to analyze database performance and investigate database queries. We can ask detailed questions like, “What are the top 5 worst performing database queries in the online boutique environment?” In the response to this question, we get a list of the database queries that have high latency, along with the number of times those queries were executed:  Detector and Alert Insights The Splunk AI Assistant can also provide insight into detectors and alerts. If we navigate to Detectors & SLOs in Splunk Observability Cloud and select critical alerts, we can see we have an active high error rate alert for our Payment service:  If we select that alert and scroll down in the alert window, we can get the Incident ID of this specific alert:  We can copy that Incident ID and then use that ID to ask the AI Assistant to explain this specific alert and tell us why it was triggered. In the response, the AI Assistant provides a summary of the alert details along with an explanation of the triggering conditions and why they were triggered:  We started investigating this specific alert by navigating to the Detectors & SLOs page view in Splunk Observability Cloud, but we could have also queried the AI Assistant to get information about active or frequently triggered detectors. The response to this query guides where to focus our attention: If these detectors are frequently triggered, it could result in service performance degradation and impact customer experience. The response to queries like this can also help us easily identify alerts that are triggered too frequently and are not actionable, which helps reduce alert noise.  Wrap up To summarize, this second post in our Splunk Observability Cloud’s AI Assistant in Action series focused on using the AI Assistant to perform deploy and release checks, analyze trace errors, investigate database query performance, and interrogate active detectors and alerts. This enabled quick insight into root cause analysis, service performance, and deployment impacts for quick issue resolution.  Stay tuned for our next post, in which we’ll use the AI Assistant to help manage organizational compliance and infrastructure costs. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  Splunk Observability Cloud’s AI Assistant Series: Identifying Unknown Unknowns AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Deploy and Release Checks YouTube video Splunk Observability Hub ... View more

Agentic AI powers the Splunk AI Assistant within the Splunk Observability Cloud interface to help you quickly and easily tap into the health of your applications and infrastructure. Simply asking the AI Assistant your observability questions using natural language does the hard work of querying your environment and returning the relevant information you need to find root cause, triage, troubleshoot, and reduce Mean Time To Resolution (MTTR). The AI Assistant can even generate the SignalFlow you need to create custom charts and dashboards.  In this Splunk Observability Cloud’s AI Assistant in Action series, we will look at how to use the Splunk AI Assistant by digging into some practical, real-world, real-time examples. We’ll explore the specific use cases of: Identifying unknown unknowns Analyzing and troubleshooting in real-time  Auditing compliance and cost  Explaining or providing feedback Onboarding new hires or new users of Splunk Observability Cloud Observability as Code This first post will explain how to access the Splunk AI Assistant, and then we’ll learn how to use it to identify unknown unknowns within our service environment.  Accessing the Splunk AI Assistant You can access the Splunk AI Assistant from Splunk Observability Cloud by selecting the AI Assistant icon located in the upper right corner of the screen: Selecting this icon opens the interactive window to chat with the AI Assistant. Once the AI Assistant pane is open, you’ll see pre-populated prompts that are a great starting point if you’re new to the AI Assistant: Here’s an example response for the first prompt, "What can you help me with?”: As you can see, the AI Assistant can access metrics, traces, and logs in Splunk Observability Cloud. It can also help you analyze detectors, alerts, incidents, and draft SignalFlow.  Identifying Unknown Unknowns As an engineer, a great way to start the work day is to interrogate the AI Assistant with a question like, “What should I know about my environment?” Whether you’re back in the office after a weekend away, jumping into an on-call rotation, or simply catching up on a recent incident to help troubleshoot, using the AI Assistant to gain quick insight into unknown unknowns is a powerful resource.  Let’s ask a question about one of our environments. We’ll prompt the AI Assistant to tell us something we might not be aware of for one of the service environments we own, the Online Boutique. We’ll ask it to point out interesting or concerning things related to this environment: Looking at the response, we can see the AI Assistant performs analysis and then identifies that the Payment service has a high error rate. It also identifies common error tags like HTTP 401 status codes and high latency concerns: We can also see that the AI Assistant analyzed the upstream and downstream dependencies of the services in our environment: It also made some recommendations based on its findings. It hyperlinked to the services involved, so we can easily investigate those services in detail by moving over to Splunk Application Performance Monitoring for further troubleshooting. In this case, we’re hyperlinked to our Payment service and can explore real-time issues directly in Splunk APM:  My team owns the Payment service, and I know that code changes were recently deployed. A great next step would be to use the AI Assistant to interrogate our system and investigate if this latest release could be causing these high error rates and latency.  Wrap up To summarize, this first post in our Splunk Observability Cloud’s AI Assistant in Action series focused on using the Splunk AI Assistant to quickly identify unknown unknowns within our environment.  Stay tuned for our next post, in which we’ll use the AI Assistant to analyze and troubleshoot in real time, investigating error rates and latency and comparing deployments and releases. Want to try out the Splunk AI Assistant for yourself? Start with a 14-day free trial! Already a Splunk Observability Cloud customer? Reach out to your account representative to enable the Splunk AI Assistant!   Resources  AI Assistant in Observability Cloud Splunk’s AI Assistant: Top 7 Use Cases for AI-Driven Observability Practical Use Cases for Splunk Observability Cloud’s AI Assistant: Identifying Unknown Unknowns YouTube video Splunk Observability Hub ... View more

Web applications unfortunately present a target rich environment for security vulnerabilities and attacks. Adding in sometimes hundreds of third party dependencies and multiple services coordinating across a variety of technologies exponentially increases the impact and likelihood of these threats. Thankfully, Splunk AppDynamics (AppD) reduces the risk of application security exposure and additionally provides many features and capabilities to optimize hybrid and on-prem application performance. To name a few, AppD offers: Features to help identify application issues in real-time and tools to then prioritize how to tackle those issues based on business metrics and impacts  Root cause analysis from anywhere in your stack with AI-driven anomaly detection Business iQ Business performance monitoring to understand how application performance directly influences critical business metrics Digital Experience Monitoring (DEM) complete with Real User Monitoring (RUM) and Synthetic monitoring for insight into how end-users interact with our applications and observe availability and performance of web apps and APIs Network monitoring to quickly isolate network performance issues with complete visibility into owned and unowned network, ISP, API, SaaS, and third-party services Application security from inside your runtime environment with the detection of vulnerabilities, blocking of attacks, and insight into potential business impacts  In this post, we’re going to dig into the capabilities provided by Splunk AppDynamics with Cisco Secure Application and check out how they help enhance application security with real-time threat detection and prevention.  Splunk AppDynamics with Cisco Application Security With Cisco Secure Application, you can easily get complete security insight and protect your applications at runtime. Typically, security scans take place before an application is deployed to production, and scans continue to take place on a monthly or quarterly basis. But Cisco Secure Application continuously detects and blocks attacks. It also reveals the potential business impacts of these attacks and vulnerabilities so you can see and fix what matters to minimize business risk and easily prioritize issues.  Cisco Secure Application features are built into Splunk AppDynamics APM Agents for:  Java .NET Core Node.js Note: to monitor the application security, you must first enable security for the application using the Cisco Secure Application dashboard.  Overview The Cisco Secure Application home page provides a comprehensive and quick view of attacks and vulnerabilities.  Business transactions provide details about the vulnerability risk of a business transaction and the overview of vulnerabilities, attacks, and applications helps you quickly identify and respond to security incidents.  Applications The Applications page provides details on monitored nodes that are registered with Cisco Secure Application for the managed applications. Here, you’ll find a centralized view of all security-related details for your applications. This real-time data allows you to detect and respond to any applications that might not have security enabled for complete application coverage.  Stats on the nodes of managed applications are also available in this view. Nodes are categorized as: Active: the number of nodes that are actively communicating with the AppDynamics Controller Supported: the number registered with the Cisco Secure Application  Ready: the number of nodes that can be successfully registered Enabled: the total number of nodes that have security enabled for the applications Secured: the number of secured nodes Trend: the number of supported, enabled, secured, active, and ready nodes against the day of the month You can also quickly navigate to the flow map by clicking on the map icon to get a holistic view of the overall application performance.  Libraries The Libraries page breaks down details for existing libraries used within applications that require remediation.  Applications that use the corresponding library along with Tier (the application tier that is vulnerable because of its relationship to the corresponding library) are also easily visible. You can again click the map icon next to an application to view the application flow map in the AppDynamics dashboard.  This view also provides quick insight into:  The Highest Cisco Security Risk Score – an estimate of exploitation based on real-time events of the vulnerability in the wild A score based on the Highest Common Vulnerability Scoring System (CVSS)  Total vulnerabilities or the number of vulnerabilities based on severity  Remediation – the recommended version of the library that can be used for remediation Status of the vulnerable libraries Libraries that are categorized as "Critical" are potentially exploitable by a bad actor, which could disrupt your services or worse give access to customer data. These items should be prioritized for immediate triage. You can also click on a specific library to view vulnerabilities for that library:  Having this information easily visible means teams can quickly identify and prioritize the most critical vulnerabilities and threats. Impacted applications, version tracking, and remediation candidates provide insight into the impact and the actionable steps to take in order to keep the application secure. Vulnerabilities The Vulnerabilities page provides a complete view of the security of your services, focusing on the types and instances of vulnerabilities instead of on each library. It includes a real-time trend graph that shows the number of both fixed and open vulnerabilities.  Vulnerabilities are displayed by severity level, and the Severity Trend chart summarizes the trend of fixing open vulnerabilities. There’s also a Days Since First Detected chart showing the number of days the vulnerability is open versus the severity.  All of this detailed vulnerability information provides fast impact analysis and in-depth information on each vulnerability along with suggested fixes through remediation candidates.  Attacks The Attacks page is critical to providing information about attacks against your services. With real-time threat detection, you can get a high-level overview of security threats and the applications that they impact.  Attacks By Outcome represents the total number of attacks based on their states, and the Top Applications chart displays the top 10 applications based on attacks per application.  Attacks are also categorized for clear breakdown of the nature of the attacks. Attack types include SQL Injection, where an attacker is attempting to use unsanitized input to execute custom SQL statements to expose or manipulate private data, Remote Code Execution (RCE), where an attacker is attempting to gain access to a system and execute custom code, and others like:  DESERIAL LOG4J SSRF MALIP You can dig into specific attack details to view things like stack traces, source details, and the impacted business transaction: Note: If you have Configure permission for Cisco Secure Application, you can create or customize policies for vulnerabilities and attacks to specify which runtime behaviors to ignore, detect, or block. Cisco Secure Application Architecture and Installation How does this all work and how do you get started with Cisco Secure Application? According to the  Secure Application Architecture documentation: Install the supported APM Agent and then add the Cisco Secure Application license. The APM-managed application runs and the APM Agent retrieves the data to send to the Controller. The Cisco Secure Application service retrieves the application, tiers, and nodes data from the Controller. The APM Agent communicates with the Cisco Secure Application service to check if the security is enabled for the application. If the security is enabled, then the agent downloads the configuration along with the policies from the Cisco Secure Application service.  Based on the configured policies, the agent sends the security events to the Cisco Secure Application service. The service collects all the data, analyzes the application behavior, and then provides the analyzed data to the Cisco Secure Application dashboard.  Wrap Up Splunk AppDynamics with Cisco Secure Application provides real-time insight into the security health of your applications. Instead of waiting for a deploy to get visibility into vulnerabilities, Cisco Secure Application provides visibility at runtime and continuously detects and blocks attacks. This helps you quickly see and fix what matters to minimize business risk and easily prioritize what to work on.  If you’re ready to get started with Cisco Secure Application, you can first check out the requirements, install the agent that supports your environment, and then contact Splunk AppDynamics sales representative or email salesops@appdynamics.com to have a Cisco Secure Application license provisioned. More info can be found in the Application Security Monitoring Getting Started docs.    Resources Splunk Delivers Unified Security and Observability to Protect Applications Splunk AppDynamics Guided Tour Splunk AppDynamics Unveils New Security and Mobile Insights Splunk AppDynamics Application Security Monitoring ... View more

We’ve already explored a few topics around observability in a Kubernetes environment -- Common Failures in a Kubernetes Environment, how to Detect and Resolve Issues in a Kubernetes Environment, and the process of Integrating Kubernetes and Splunk Observability Cloud. But even though the process of setting up the OpenTelemetry Collector and exporting telemetry data to a backend observability platform like Splunk Observability Cloud can be completed in a few short steps, these initial setup steps don’t account for what happens when new workloads are spun up. Kubernetes is known for its dynamic nature, and observability needs to move right along with the ever-changing environment. In this post, we’ll see how monitoring Kubernetes workloads is currently done. Then we’ll check out how the new Kubernetes annotation-based discovery for the OpenTelemetry Collector makes it even easier. Monitoring Workloads: The Traditional Approach Monitoring dynamic workloads typically requires manual definitions of conditional configurations. Engineers define configuration “templates” based on environment conditions that adjust the monitoring configuration based on what’s automatically discovered in the environment. For example, to dynamically discover and monitor nginx pods that are deployed on the cluster, a receiver_creator definition is required:   This configuration is enabled when a pod is discovered via the Kubernetes API that exposes port 80 (HTTP port) and matches the nginx keyword. This approach works, but it requires a defined Collector configuration for each workload type. If new workload types are added to the environment, the configuration needs to be updated and redeployed, which can be time-consuming – especially if different teams are responsible for different aspects of the deployment. Additionally, it relies on the name of the pod, limiting flexibility to name workloads after what they actually do, making it harder to troubleshoot and diagnose problems. Monitoring Workloads: The New Approach The recently added Kubernetes annotation-based discovery for the OpenTelemetry Collector now allows for the dynamic configuration of monitoring workloads by adding annotations directly to their pods or namespaces. The K8s observer detects and reports information about objects appearing in the cluster to the Collector’s receiver_creator and automatically adjusts its configuration to monitor the annotated workloads. The annotations define the receiver that’s used to scrape telemetry data.  Here’s an example pod annotation that tells the OpenTelemetry receiver_creator to scrape metrics and enables discovery on the pod:    Finally, the discovery functionality must be enabled in the receiver_creator:   Once the deployment changes are applied and the Collector is restarted, you’re done! Adding a new nginx workload? No problem. Annotate your pods as shown above to dynamically configure monitoring for your new workload. It doesn’t matter what the pod is named and enables you to be more declarative about what is collected and how. Wrap up The amount of configuration maintenance required significantly decreases thanks to this new annotation-based discovery for Kubernetes and the OpenTelemetry Collector. Kubernetes observability becomes as dynamic as the environment it monitors, and configuration of dynamic workload discovery becomes a seamless experience. Already using the OpenTelemetry Collector in Kubernetes and want to try out this new feature? Want to export your automatically discovered Kubernetes workloads and their telemetry data to a unified observability platform? Check out Splunk Observability Cloud’s free 14-day trial. Resources Generate receiver configurations from provided Hints Kubernetes Monitoring and Alerting Made Easy with Splunk Observability Cloud and OpenTelemetry  Kubernetes Monitoring: The Ultimate Guide How to Monitor Kubernetes with Splunk Infrastructure Monitoring ... View more

Culture extends beyond work experience and coffee roast preferences on software engineering teams. Team culture also includes opinions on ticket creation/completion, merge request reviews, on-call rotations, and incident response. When it comes to incident response, a Don’t Repeat Incident (DRI) team or company culture can be hugely beneficial in maintaining a reliable code base, a positive user experience, and overall team happiness.  What is DRI? When an incident occurs, on-call team members are notified of triggered alerts. These individuals are often pulled into an incident response role to troubleshoot and resolve the incident. Traditionally, once the incident is resolved, the work of the on-call team members is over. Contrast this with a DRI approach: when the incident is successfully mitigated, the work is not over. Post-incident, work needs to be tasked out and prioritized to address any code changes or safeguards that need to be put in place to prevent the same incident from occurring again in the future – every incident results in the creation of one or more actionable tickets or issues. Sidenote: post-incident reviews (or postmortems) are also a great idea and serve as a retrospective around the incident to discuss why it occurred, how it could have been prevented, who was impacted, and how it will be prevented in the future.  When this whole process works, it helps create a proactive culture and reduces incident frequency. However, manual steps always leave room for error – a person forgets to create a ticket, a person forgets to follow up, and soon alerts for errors that have been seen before are firing yet again. Taking manual intervention out of the equation helps guarantee that tickets are created, added to the right repositories or boards, and prioritized to prevent incidents from repeating. How can technology help us with this? Cue webhooks. Webhooks Webhooks are a great way for systems to communicate when specific events occur. They can be used for a range of interactions – building out CI/CD pipelines, real-time data sharing like when sending confirmation emails after online order completion, initiating two-factor authentication at login, etc.   Because of their lightweight efficiency, many platforms support webhooks – Gmail, Slack, GitHub, Jira – the list is long. In this post, we’ll look at how Splunk Observability Cloud can help with the DRI practice described above to reduce incidents, increase code resiliency, and remove the need for manual intervention with webhooks.  Splunk Observability Cloud & Webhooks Your environment is unique, and Splunk Observability Cloud provides many integrations out-of-the-box (Slack, Jira, ServiceNow) that may suit your incident response needs. However, in situations where your environment needs aren’t met, custom webhook integrations have your back.  Say we keep track of our product’s code issues using GitHub issues. We can create a webhook integration in Splunk Observability Cloud to track active incidents by automatically opening an issue anytime an alert fires. Again, this helps ensure that human eyes land on the root cause and mitigation work is identified and recorded in an issue to make sure the incident does not repeat.   Let’s look at how we can go about setting up a GitHub webhook in Splunk Observability Cloud.  Configure a GitHub Webhook in Splunk Observability Cloud First, we’ll navigate to Data Management in Splunk Observability Cloud and search for the Webhook integration:    We can follow along with the guided setup and configure our GitHub connection:    Select Next to customize the auto-populated payload (but first, notice how much data you can send and act on in the remote system):    We’ll update the fields to match those required by the GitHub API to create a new issue and use the messageTitle and description variables to populate our issue:  After selecting Next we can review and save our webhook:  With our GitHub webhook saved, we next need to add it as a notification recipient on the desired detectors.  Note: to add a webhook as a detector recipient, you must have administrator access.  We can configure webhooks by editing the detector of choice:  We can add Alert recipients and select Webhook:   Then select our newly created GitHub Issue Webhook as the recipient and activate our updated alert notification:    Note: our GitHub webhook hits the GitHub REST API to create issues and these are scoped to the repository specified in the request URL – in our case, the Worms in Space repository. That means we would want to make sure the detector we attach this webhook to is related to that code repository specified in the URL. Thankfully, when we create detectors and alert rules, we can easily scope them to specific services.  That’s it! When an alert rule is triggered (and in this case when an alert is cleared or resolved), we’ll see a new issue automatically appear in our repo’s GitHub issues:  This is a simple example, but you can imagine how you could integrate this with a CI/CD system, for example, to monitor a critical metric, fires off a webhook that runs a script to determine if there’s been a recent release, and rolls back the release if the metric goes out of bounds. Exact examples vary a lot because there are so many systems out there, but the possibilities are really endless. Wrap up Now that our alerts create issues in our GitHub repository, we can prioritize and resolve their root cause and eliminate repeat incidents. Rather than being pulled away by reoccurring alerts, our focus can stay on delivering high-priority code, our applications can stay resilient, and our users can stay happy.  Want to accomplish something similar with Jira? Check out the Splunk Observability Cloud built-in Jira integration to easily connect Jira projects and create issues based on Splunk Observability Cloud alerts. Interested in building out a custom webhook? That works too! Once you’ve built a custom webhook, follow the same steps above so it can listen for and receive Splunk Observability Cloud alert notifications.  Don’t yet have Splunk Observability Cloud? Try it free for 14 days! Resources Send alert notifications to a webhook using Splunk Observability Cloud Send alert notifications to Jira using Splunk Observability Cloud Send alert notifications to services using Splunk Observability Cloud ... View more

Logs and Metrics Logs are generally unstructured text or structured events emitted by applications and written to files. Logs are essential to system visibility, especially when investigating unpredictable states of the system. Logs (ideally) give you data on exactly why something isn’t working. However, a logs-only approach becomes slow, expensive, and unsustainable as systems and the data they produce grow.   Metrics are data points that represent numbers, counters, or gauges that provide visibility into a system — error count, average CPU utilization percentage, throughput, etc. combined with a time stamp. Metrics are great for quick and reliable big-picture insight into system performance and can help teams quickly identify predictable system error states. But while out-of-the-norm metrics might indicate a problem within an application environment, it’s hard to pinpoint the root cause by looking at metrics alone.  Using a combination of logs and metrics provides many of the details needed to troubleshoot issues. So if we need both logs and metrics, what’s all this business about converting logs to metrics?  Why convert logs to metrics? In many systems, including Splunk, metric data points are stored in a format that provides faster search performance and more efficient data storage. This means log data that would be better represented by high-level, quantitative metrics rather than detailed text can be converted to metrics to help set alerting thresholds, reduce storage costs, and decrease the time it takes to troubleshoot issues. Rather than searching through logs of HTTP status codes and trying to parse patterns or identify outliers from the log text, we can convert status codes to metrics and easily view anomalies or even alert on them.  So why not just start off reporting these metrics? Maybe you’re already logging system data but are unable to alter the code base to instrument metric data – or perhaps you’re using a  commercial off-the-shelf (COTS) application that only emits logs. Maybe you need a low-barrier-to-entry way to gain deeper insight into your system with better search performance and lower data storage costs. Maybe certain teams need access to specific metric data but aren’t permitted to access the full details produced in logs. No matter the reason, pulling metric data from logs provides a higher-level system overview, a faster way to query data, and the ability to set alert thresholds on that metric data.  Convert logs to metrics with the OpenTelemetry Collector So how do we convert logs to metrics? Many observability platforms provide in-app ways to convert logs to metrics, so depending on which backend provider you use, the process can look different. In Splunk Enterprise, for example, ingest-time log-to-metrics conversion can be accomplished through Splunk Web or through configuration files.  For a quick and easy vendor-agnostic way to start reporting the metrics you need from the logs you already have, the OpenTelemetry Collector can be installed and configured to convert logs to useful metric counts. The Collector can be thought of as a pipeline to ingest logs from your application, process and extract metrics, and then send the metrics to an observability backend. Using the OpenTelemetry Collector approach allows us to:  Preprocess logs and offload the logs to metrics conversion from logging backends to the Collector  Send our metrics to locations outside of logging backends (like Splunk Observability Cloud) Reduce the need to store certain logs in logging platforms – instead, create the metrics and drop the logs or send them to low-cost archives Set our application up for future standardization by laying the groundwork for using OpenTelemetry Let’s take a look at how to do this.  Configuring the OpenTelemetry Collector Since we’re working with Splunk, we’ve installed the Splunk Distribution of the OpenTelemetry Collector. If you want to follow along and install the Collector yourself, you can work through the steps in the Splunk documentation to get started with the Splunk Distribution of the OpenTelemetry Collector.   With the Collector installed, we can configure the otel-collector-config.yaml file. First, we’ll update the receivers block to ingest our application’s log data:  This receiver sends our log data into the Collector just like it would to an HTTP Event Collector so no application code changes are required.  Next, we’ll need a connector. Connectors act as both an exporter and a receiver – they both consume and emit data. We’ll use a connector to consume our log data and then emit that data as a metric: The count connector is used to create a http_500_errors metric that counts any log events where the status code is equal to 500. The defined attributes will additionally create any dimensions we’d like to include on this count metric.  Note: we could also use the sum connector to convert our logs to sum attribute values. The connector you use depends on the type of metric you need and what you want the metric to represent. Jeremy Hicks wrote a super great, in-depth blog post on this called Introduction to the OpenTelemetry Sum Connector. Here’s a quick example of what using the sum connector would look like:  We’re going to move forward with the count connector example. We’ll want to export this new count metric to Splunk Observability Cloud, so we’ll also include an exporter for that:  If we wanted to send our logs to an HTTP Event Collector, we could also add another exporter here. In the future, we might even want to exclude exporting the logs we convert to metrics by including a filter processor here in our configuration.  Finally, we need to add these to the service pipelines: Here, we’ve configured our pipeline to receive logs from Splunk Enterprise, receive metrics from our count/http-error-500 connector, and export metrics to our otlp endpoint, aka Splunk Observability Cloud.  We can then save this configuration, start/restart the OpenTelemetry Collector, and search for our new metrics from the Metric Finder in Splunk Observability Cloud:  Wrap up Converting logs to metrics gives us a high-level view of our system and allows us to create dashboards, charts, and detectors to proactively detect and alert on anomalies. We can then use this information to dig into our logs for deep insight into the exact lines of code that are failing for super-fast issue resolution.  If you’re ready to convert your logs to metrics and visualize your metric data in an observability platform with unified, full-system visibility, start with a Splunk Observability Cloud 14-day free trial.  Resources Logs vs Metrics: Pros, Cons & When to Use Which From Webhook to Log and Log to Metric: OpenTelemetry Magic! Set up ingest-time log-to-metrics conversion in Splunk Web ... View more

Agent Saturation What and Whys In application performance monitoring, saturation is defined as the total load on a system or how much of a given resource is consumed at a time. If saturation is at 100%, your system is running at 100% capacity. This is generally a bad thing. Agent saturation is a similar concept. It represents the percentage of available system resources currently being monitored by an observability agent. 100% agent saturation means 100% of available resources are instrumented with observability agents, which is a great thing. When it comes to observability practices, agent saturation is how well a system is instrumented and can be represented by:  ( instrumented resources / total resources ) x 100 Because greater visibility into system health and performance means proactive detection of issues, improved user experience, more efficient troubleshooting, decreased downtime, and countless other pluses, 100% agent saturation is the ultimate goal.  So why doesn’t everyone get to 100% agent saturation for full system observability and magical unicorn application visibility status? It’s challenging! Setting up observability agents across distributed applications and environments takes time. In ephemeral, dynamic systems already integrated into existing solutions, it can just be too much of a lift.  But the good news is that if you’re already using Splunk (maybe for logging and security) there are quick and easy ways to improve your system observability. In this post, we’re going to look at how to leverage the Splunk Add-on for OpenTelemetry Collector to gain a quick win when it comes to improving agent saturation.  Improve Agent Saturation with the Splunk Add-on for OpenTelemetry Collector For Splunk Enterprise or Splunk Cloud Platform customers who ingest logs using universal forwarders, you can quickly improve agent saturation and deploy, update, and configure OpenTelemetry Collector agents in the same way you do any of your other technology add-ons (TAs). The Splunk Add-on for OpenTelemetry Collector leverages your existing Splunk Platform and Splunk Cloud deployment mechanisms (specifically the universal forwarder and the deployment server) to deploy the OpenTelemetry Collector and its capabilities for increased visibility into your system from Splunk Observability Cloud. The add-on is a version of the Splunk Distribution of the OpenTelemetry Collector that simplifies configuration, management, and data collection of metrics and traces. This means OpenTelemetry instrumentation will out-of-the-box exist anywhere the universal forwarder is present for logs and security use cases, making it easier to instrument systems quickly and gain visibility into telemetry data from within Splunk Observability Cloud. This comprehensive system coverage also comes with out-of-the-box Collector content and configuration with Splunk-specific metadata and optimizations (like batching, compression, and efficient exporting), all preconfigured. This means that you can get answers using observability data faster, saving you time and effort. Prerequisites for using the Splunk Add-on for OpenTelemetry Collector include:  Splunk Universal Forwarder (version 8.x or 9.x on Windows or Linux) Splunk Observability Cloud Splunk Enterprise or Splunk Cloud or deployment server as forwarders (Optional) Install the deployment server if you plan to use it to push the Collector to multiple hosts Getting started with the Splunk Add-on for OpenTelemetry Collector The Splunk Add-on for OpenTelemetry Collector is available on Splunkbase similar to other TAs, and you can deploy it alongside universal forwarders using existing Splunk tools like the deployment server.  We have a Linux EC2 instance we’re going to be instrumenting, but we first need to download the Splunk Add-On for OpenTelemetry Collector from Splunkbase:  We’ll unzip the package and then create a local folder and copy over the config credential files: In Splunk Observability Cloud, we’ll get the access token and the realm for our Splunk Observability Cloud organization:  Your organization's realm can be found under your user’s organizations:  Next, we set these values in our /local/access_token file: We then need to make sure the Splunk Add-On for OpenTelemetry folder (Splunk_TA_otel) is in the deployments app folder on the deployment server instance: We’ll then move over to the deployment server UI in Splunk Enterprise to create the Splunk_TA_otel server class and add the relevant hosts along with the Splunk_TA_otel app. Once the TA is installed make sure you check both Enable App and Restart Splunkd and select Save: That’s it! If we now navigate to Splunk Observability Cloud, we’ll see the telemetry data flowing in from our EC2 instance:  Wrap up Increasing agent saturation and improving observability for comprehensive system insight can be quick and easy. Not sure how you’re currently doing in terms of agent saturation? Check out our Measuring & Improving Observability-as-a-Service blog post to learn how to set KPIs on agent saturation. Ready to improve your agent saturation? Sign up for a Splunk Observability Cloud 14-day free trial, integrate the Splunk Add-on for OpenTelemetry Collector, and start on your journey to 100% agent saturation.  Resources Splunk Add-On for OpenTelemetry Collector Differences between the OpenTelemetry Collector and the Splunk Add-on for OpenTelemetry Collector Get started with Splunk Observability Cloud ... View more

Introduction In observability monitoring, static thresholds are used to monitor fixed, known values within application environments. If a signal goes above or below a static threshold or is within or outside a specified range, an alert will fire. Static thresholds are quick to configure and can provide helpful insight into system stats, but there are downsides to using them too frequently. There’s a time and a place for static thresholds, and in this post, we’ll look at when to use static thresholds, when not to use them, and alternatives to static thresholds.  When to use static thresholds Static thresholds work well for situations where there are “known knowns” – the predictable cases that you can anticipate – and when there’s a static range of “good” and “bad” values. Here’s an example of a CPU utilization detector with an alert rule that triggers when CPU is above 90% for a 5 minute duration:    As a side note, adding durations on such alert rules is important to avoid over-alerting on transient spikes. Without a set duration, every CPU spike over 90% would trigger an alert, as we can see when we try to configure the same condition without a duration:  The estimated alert count for this alert rule is 11 alerts in 1 hour – aka too much alert noise.  Boolean conditions, like monitoring synthetic test failures, are a great case for static thresholds. Alerting when a synthetic check fails indicates issues with site availability that should be addressed immediately. This detector alerts on a synthetic test when uptime drops below the 90% static threshold:    You can also use static thresholds to monitor Service Level Objectives/Service Level Agreements/Service Level Indicators (SLO/SLA/SLI). If you have an SLO of 99.9% uptime, you’ll want to be alerted anytime your availability approaches that threshold. Here’s an SLO detector that alerts if latency passes the 99.99999% threshold:    While configuring static thresholds on SLOs is possible, within Splunk Observability Cloud it’s recommended to manage SLO alerts using error budgets.  Static thresholds are most appropriate when working with fixed and critical metrics with clear failure conditions, e.g. HTTP error response codes spiking, response time increasing for a certain period of time, error rates above a certain percentage for a certain amount of time. If you’re just starting out on your observability journey, using static thresholds is also a great way to capture baseline metrics and gain insight into trends so you can fine-tune and adjust your detectors and alerts.  When not to use static thresholds As we saw in the CPU detector above, alerting on static thresholds can lead to a lot of alert noise if not used correctly. To create good detectors, we need to alert on actionable signals based on symptoms, not causes (we have a whole post on How to Create Good Detectors).  In dynamic system environments that include autoscaling and fluctuating traffic, static thresholds might not indicate actual system failures. Setting static thresholds on pod CPU in a Kubernetes environment, for example, could indicate increased load, but might not indicate a problem – pods could autoscale to handle the increase just fine.  Note: monitoring the CPU static threshold combined with pod queue length could provide the additional context needed to create an actionable detector.  When there are periods of traffic spikes – Black Friday for e-commerce sites, lunchtime for a food delivery site – setting static thresholds can lead to an increase in false alarms. Applications with such variable traffic fluctuations might not benefit from alerting on static thresholds alone.  Dynamic resource allocation and variable usage patterns can make using static thresholds in isolation tricky, but thankfully, alternative approaches can help. Alternatives to static thresholds For the situations mentioned above, along with those where we might not know all of our application’s failure states (unknown-unknowns), alternative detector thresholding or a hybrid approach can work best. Alternatives include but definitely aren’t limited to:  Ratio-based thresholds – instead of alerting on 500 MB of memory used, use a threshold of 80% of total memory for a specific duration of time Combining static thresholds with additional context – high CPU + pod queuing or error rate or latency  Sudden change detection – alerts on sudden changes to specified conditions like number of logins, response time, etc.  Historical anomaly detection to baseline environments and alert on deviations from trends – alerting on latency that deviates from historical trends Combining out-of-the-box approaches with custom thresholds can help you build a resilient monitoring solution to keep your applications running smoothly and your users happy.  Wrap up  For predictable, critical metrics static thresholds are a great alerting option. For situations when static thresholds aren’t appropriate, there are thankfully additional solutions that can help. To start, check out Splunk Observability Cloud’s out-of-the-box alert conditions here and explore what might work best for your unique application environment. Don’t yet have Splunk Observability Cloud? Try it free for 14 days!  ... View more

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth analysis on your incoming telemetry data. SignalFlow is the computational backbone that powers all charts and detectors in Splunk Observability Cloud, but the statistical computation engine also allows you to write custom programs in the SignalFlow programming language (modeled after and similar to Python). Writing SignalFlow programs definitely isn’t required for a complete observability practice, but if you want to perform advanced computations on your Splunk Observability Cloud data, SignalFlow is your friend. In this post, we’ll explore the whys and hows of SignalFlow so you can run these computations, aggregations, and transformations on your data and stream the results to detectors and charts for custom and in-depth observability analytics. Why use SignalFlow? Most Splunk Observability Cloud use cases don’t require complex computations. Out-of-the-box charts and detectors make building a complete observability solution easy. But sometimes there is a need for more detailed and advanced insight. SignalFlow can be used to: Define custom behavior or conditions for fine-tuned control over your monitoring so you can tailor your charts and detectors to your specific needs Aggregate metrics from different applications, cloud providers, or environments to unify data Troubleshoot by correlating metrics across many different sources – using SignalFlow during an incident helps provide deep, real-time investigation into root cause Detect trends over time or compare historical data to increase resiliency, reduce downtime, and capacity plan – i.e. correlate resources with user activity over time to optimize resource allocation Stream metric data to background analytics jobs – i.e. execute computations across a population over time Create reports or visualizations in third-party UIs so you can stream data out of Splunk Observability Cloud – i.e. in a service provider use case you can stream data to your own UIs and expose it to your customers Correlate business metrics with infrastructure and/or application metrics to understand how performance impacts customers – i.e. how does latency impact customer renewal rates There are many reasons why you might need to tap into SignalFlow; generally, if you need customized analytics, SignalFlow is the answer. So let’s see how it works! How to use SignalFlow You can define SignalFlow queries directly in the Splunk Observability Cloud UI or programmatically using the SignalFlow API. If you open up or create a chart in the UI, you’ll see the chart builder view:  If you select View SignalFlow, you can dive right into the SignalFlow that powers the chart and use it as a template for additional programs:  The same is true for detectors. If you open up a detector, you can select the kebab icon to Show SignalFlow:    SignalFlow programs outside of the Splunk Observability Cloud UI typically live within code configurations for detectors and dashboards (see our Observability as Code post). When you create a chart or detector using the API, you can specify a SignalFlow program as part of the request. Here’s an example of defining a detector using Terraform, where the program_text is the SignalFlow program:  You can also use the Signalflow API to run programs in the background and receive the results asynchronously in your client. Let’s take a look at some SignalFlow functions and methods we can use to build out charts and detectors.  SignalFlow Functions and Methods in Charts Most SignalFlow programs begin with a data() block. The data function is used to query data and is the main way to create stream objects, which are similar to a time-parameterized NumPy array or pandas DataFrame. Queries can run against both real-time incoming streams and historical data from the systems you monitor. In SignalFlow, you specify streams as queries that return data. Here’s a template for the data function:  We can expand on the data function in many ways. For example, here’s what it would look like to query for the CPU utilization metric and filter by host: We can also add/chain methods or functions to our data block. Here are examples of using the mean method to look at mean CPU utilization, mean CPU utilization by Kubernetes cluster, and mean CPU utilization over the last hour:  Operations like mean, variance, percentile, exclude, ewma, timeshift, rate of change, standard deviation, map(lambda), and others are available as methods on numerical streams. Here’s an example where our data stream, signal, is the CPU utilization with a filter of host, and we can add functions to timeshift by a week and two weeks, and then find the max value:  Comparing the max CPU utilization for two separate time series can’t actually be accomplished using the chart plot editor in the Splunk Observability Cloud UI, so this is an instance of where using SignalFlow is necessary.  To actually output these stream results to a chart, we need to call the publish() method:  We’ve now built out a chart using SignalFlow 🎉! We can also do this with detectors – read on. SignalFlow Functions and Methods in Detectors Detectors evaluate conditions involving one or more streams, and typically compare streams over periods of time – i.e. disk utilization is greater than 80% for 90% of the last 10 minutes. When building detectors using SignalFlow, we still start with our data streams, and then transform our data streams using boolean logic:  Note: when setting static thresholds in the UI, thresholds can only be greater than or less than. But as you can see with SignalFlow, we can specify greater than or equal to static thresholds.  We can use these when statements on their own or combined with and, or, not statements to publish our alert conditions and build out our detectors:  The detect streams in this example are similar to data streams. Detect streams turn our boolean statement – when our signal is greater than 90 for 1 minute – into an event stream. When this statement is evaluated as true, an event will fire and be published to an event stream. This is what triggers an alert.  Note: event streams are evaluated and published in real time as metrics are ingested, enabling you to find problems faster and speed up your MTTD. Every publish method call in a SignalFlow detect statement corresponds to a rule on the Alert Rules tab in the Splunk Observability Cloud UI. The label inside the publish block is displayed next to the number of active alerts in the Alert Rules tab:  You can create your detectors using the SignalFlow API, but if you want to use SignalFlow to build detectors directly in the Splunk Observability Cloud detector UI, you can append /#/detector/v2/new to your organization URL to do so:  Wrap up While working with SignalFlow is not required, it can help customize and advance your observability practice. A great place to start is editing the SignalFlow for existing charts and detectors in the Splunk Observability Cloud UI or using observability as code with SignalFlow programs. In no time, you’ll be building out SignalFlow program background jobs and streaming customized analytics to meet all your observability and business needs.  New to Splunk Observability Cloud? Try it free for 14 days!  Resources SignalFlow and analytics Analyze data using SignalFlow Up Close Monitoring with SignalFlow Intermediate to advanced SignalFlow ... View more