Manual Instrumentation with Splunk Observability Cloud: The What and Why

[CaitlinHalla]

If you've ever worked with distributed systems, you’ve likely felt the pain of a frontend throwing errors, while your backend is processing requests slower than usual, and somewhere between your resolvers and database queries something unknown is going wrong. But what are the specific issues, where exactly are they occurring, and why did they happen in the first place? This is exactly the kind of insight observability provides, and the benefits of having such visibility into systems are endless. Faster insight into problems means proactive prevention of disruptions and performance issues. Visibility into the root cause of issues leads to quick pinpointing of the exact lines of code that need to be fixed, which means the impact to our users is minimal or non-existent, and our customers stay happy.  

Observability platforms make it easy to gain visibility into application ecosystems through unified, intuitive UIs, and automatic instrumentation. Collecting and exporting the data required for this insight is often automatic through tools like automatic discovery and zero-code instrumentation. But even when automatic instrumentation is available, there are times when only you know what really matters for your application’s success. Adding in manual or custom instrumentation can provide the full picture when it comes to specific application logic or insight related to business-specific data.

Manual instrumentation can be an invaluable tool in your observability toolkit, and in this series, we're going to dig into it to see how we can use it to easily build observability from scratch. We’ll look at a real application and take it from zero observability to fully instrumented, commit by commit.

In this first post, we’ll learn about what manual instrumentation is and why you need it. Then, in subsequent posts, we'll look at a real, un-instrumented application, and go step-by-step through the process of manually instrumenting it.  

What Is Manual Instrumentation? 

Automatic instrumentation seems almost magical. Install an agent or library, restart your service, and poof -- metrics, logs, and traces start flowing into your observability platform. But what about the parts of your code that don’t get picked up automatically? The times where your services aren’t written in automatic instrumentation-supported languages? Or the times when you need more context than what’s provided by auto-instrumentation? That’s where manual instrumentation comes in.

Manual instrumentation is the process of explicitly adding observability code to applications to capture custom telemetry data. Observability platforms often provide auto-discovery, zero-code implementations – including the use of which was recently donated to the community by Splunk – that fully instrument systems. But not all languages and services are supported when it comes to auto-instrumentation. Even in cases where auto-instrumentation is possible, we sometimes need more detail. Custom instrumentation around business-specific telemetry data and specific code or custom attributes provides invaluable business context and captures the nuances for our unique applications. In these cases, manual instrumentation adds a ton of value.  

Unlike with automatic instrumentation, with manual instrumentation, you take control and decide exactly what and how to instrument. This means that even if your observability platform doesn’t automatically provide visibility into your code base, and your application starts throwing errors at 2 AM, you’ll still be notified about the failures, and you'll know exactly why and where errors are occurring. 

Why Manually Instrument?

In cases where automatic-instrumentation for a language or service isn’t available or where out-of-the-box telemetry data isn’t capturing the full application picture, manual instrumentation is a necessity. Manual instrumentation offers full-stack observability, business logic insight, and custom metrics and attributes. Let’s look at each of these benefits.

Full-Stack Visibility 

Manual instrumentation helps complete the full application picture for complete visibility into performance and issues. With automatic instrumentation you can see things like degraded endpoint performance, but it won't always tell you which part of your custom sorting algorithm is causing bottlenecks. With manual instrumentation, you can add instrumentation around the specific code to identify exactly where time is being spent. Is the bottleneck in the data transformation? The third-party API call? Now you'll know. Manual instrumentation lets you add these detailed breadcrumbs throughout your code, capturing variable states, decision outcomes, and execution paths. It's like leaving yourself detailed notes about what your code was doing at each step.

Business Logic Insight

Your application isn't just a collection of HTTP endpoints and database calls. It's a carefully orchestrated system of business rules, calculations, and decision points. Manual instrumentation lets you trace through complex workflows like order processing, fraud detection algorithms, or multi-step approval processes. When something goes wrong with a customer's order at 2 AM, you want to know exactly which business rule failed and why. Manual instrumentation allows you to capture this business-specific insight.

Custom Metrics and Attributes   

Every application has its own set of unique and important metrics. Maybe it's the number of items in a shopping cart, the complexity score of a generated report, or the cache hit rate for your custom caching layer. Manual instrumentation lets you capture these application-specific metrics and add custom attributes that provide context, like customer tier, feature flags, or regional data. 

Wrap up

For cases where automatic instrumentation isn’t possible or doesn’t provide all the features you need, manual instrumentation gets you critical visibility to keep your applications up and running. When automatic instrumentation is available, it gets you to broad coverage quickly, but adding in manual instrumentation gets you deeper insights.

Manual instrumentation helps you go beyond what’s possible with automatic instrumentation, so you have the power to see exactly what's happening, when it's happening, and why it's happening – turning mysterious production issues into clearly diagnosed problems with obvious solutions.

Join us in our next post to see how you can easily go from zero observability to full visibility with manual instrumentation.

Ready to take control of your observability today? Start with OpenTelemetry documentation and examples to begin instrumenting your application. If you need a platform where you can visualize all your telemetry data in one place, Splunk Observability Cloud's 14-day free trial is a great way to begin your observability journey.

Resources

• Instrument back-end applications to send spans to Splunk APM
• Instrumentation methods for Splunk Observability Cloud
• Automatic discovery of apps and services
• OpenTelemetry Language APIs & SDKs