https://community.splunk.com/t5/Community-Blog/Manual-Instrumentation-with-Splunk-Observability-Cloud-How-to/ba-p/752186 <P>Manual instrumentation is an invaluable observability tool. In this series, we dig into manual instrumentation to see how we can use it to easily build out observability from scratch. In our first post, <A href="https://community.splunk.com/t5/Community-Blog/Manual-Instrumentation-with-Splunk-Observability-Cloud-The-What/ba-p/751816" target="_blank">Manual Instrumentation with Splunk Observability Cloud: The What and Why</A>, we learned about what manual instrumentation is and why you need it. In this post, we'll look at a real, un-instrumented full-stack application and go step-by-step through the process of manually instrumenting its backend, commit by commit.</P><P><FONT size="6">Preface: SDK-based Automatic Instrumentation</FONT></P><P>Before we jump in, we want to call out that while we will be using manual instrumentation to create custom telemetry data, we’ll also be using <A href="https://opentelemetry.io/docs/languages/" target="_blank">OpenTelemetry’s SDK-based automatic instrumentation</A> to hook into our web framework, HTTP requests, and database so we don’t have to manually create spans for framework operations.&nbsp;</P><P>A true <A href="https://help.splunk.com/en/splunk-observability-cloud/manage-data/available-data-sources/supported-integrations-in-splunk-observability-cloud/apm-instrumentation/instrumentation-methods" target="_blank">zero-code</A> example of automatic instrumentation would require no code changes, but since zero-code auto-instrumentation doesn’t exist <SPAN>(yet) </SPAN>for our application’s <A href="https://elixir-lang.org/" target="_blank">Elixir</A> backend, we’ll use SDK-based automatic instrumentation for framework operations and full manual instrumentation for our business logic instrumentation.&nbsp;</P><P><FONT size="6">Example Application Overview&nbsp;</FONT></P><P><STRONG>GitHub repository: </STRONG><A href="https://github.com/splunk/evangelism-public/tree/main/manual-instrumentation" target="_blank">https://github.com/splunk/evangelism-public/tree/main/manual-instrumentation</A></P><P>Our un-instrumented example application is called Worms in Space. It consists of a <A href="https://react.dev/" target="_blank">React</A> frontend with an Elixir backend and provides a full <A href="https://graphql.org/" target="_blank">GraphQL</A> API. Worms in Space allows customers (who happen to be worms) to schedule spacewalks in… space.&nbsp;</P><P>Space worms can use the UI or GraphQL API to:</P><UL><LI>View available time slots</LI><LI>Schedule space walks during predefined or custom time slots</LI><LI>View all scheduled space walks</LI><LI>Cancel space walks</LI></UL><P>Here’s what the frontend UI looks like:&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_0-1756155301408.jpeg" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40066i8F8800A65AFEB791/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_0-1756155301408.jpeg" alt="CaitlinHalla_0-1756155301408.jpeg" /></span></P><P>And here you can interact with the GraphQL API via <A href="https://github.com/graphql/graphiql/tree/main/packages/graphiql#readme" target="_blank">GraphiQL</A>:&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_1-1756155301410.png" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40065i62156F2FB7280FE9/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_1-1756155301410.png" alt="CaitlinHalla_1-1756155301410.png" /></span></P><P>If you'd like to follow along on GitHub commit-by-commit, check out the GitHub repository, otherwise, you can read on to follow each commit here.</P><P><FONT size="6">Backend Instrumentation Implementation</FONT></P><P>Again, our example application is an Elixir backend with a GraphQL API for scheduling spacewalks. Even if you aren’t using Elixir or GraphQL, you can still use the following general steps as a guide to instrument your own backend application:</P><UL><LI>Add the necessary OpenTelemetry dependencies</LI><LI>Configure OpenTelemetry</LI><LI>Initialize instrumentation</LI><LI>Add custom telemetry data</LI></UL><P>Let's walk through each of these steps commit-by-commit to instrument our Worms in Space backend.&nbsp;</P><P><FONT size="5">Step 1: Add OpenTelemetry Dependencies</FONT></P><P><STRONG>Git commit:</STRONG> <A href="https://github.com/splunk/evangelism-public/commit/5f926245" target="_blank">5f926245</A></P><P>The first thing we need to do within our backend application is add the OpenTelemetry dependencies to our dependency file, in our case our <FONT face="andale mono,times">mix.exs</FONT> file:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_2-1756155301412.png" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40064i52B7FE47C9496994/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_2-1756155301412.png" alt="CaitlinHalla_2-1756155301412.png" /></span></P><P>The list of dependencies can be found in the OpenTelemetry Language APIs &amp; SDKs doc under the <A href="https://opentelemetry.io/docs/languages/erlang/getting-started/#dependencies" target="_blank">Elixir/Erlang dependencies section</A>. Since we’re using the core OpenTelemetry SDK, we’ll need to include OpenTelemetry, the OpenTelemetry API, and the OpenTelemetry exporter. Our Elixir application is using <A href="https://www.phoenixframework.org/" target="_blank">Phoenix</A> for HTTP requests, so to auto-instrument our controllers, plugs, and requests, we’ll need to include the OpenTelemetry dependency for Phoenix. Phoenix uses <A href="https://github.com/ninenines/cowboy" target="_blank">Cowboy</A> as its default HTTP server, so we’ll also include the OpenTelemetry Cowboy dependency so we can low-level HTTP requests. Our application uses <A href="https://github.com/elixir-ecto/ecto" target="_blank">Ecto</A> for database interactions, so we’ll include the OpenTelemetry Ecto dependency, and then &nbsp;GraphQL operations in Elixir are resolved via <A href="https://github.com/absinthe-graphql/absinthe" target="_blank">Absinthe</A>, so we’ll need the OpenTelemetry Absinthe dependency, as well. Finally, the <A href="https://opentelemetry.io/docs/specs/otel/protocol/exporter/" target="_blank">OTLP exporter</A> is required for sending telemetry data to Splunk Observability Cloud.&nbsp;</P><P><FONT size="5">Step 2: Configure OpenTelemetry SDK and OTLP Exporter</FONT></P><P><STRONG>Git commit:</STRONG> <A href="https://github.com/splunk/evangelism-public/commit/e45ee707" target="_blank">e45ee707</A></P><P>Next, we configure the <A href="https://opentelemetry.io/docs/languages/php/sdk/" target="_blank">OpenTelemetry SDK</A> in our <FONT face="andale mono,times">config/runtime.exs</FONT> file:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_3-1756155301419.png" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40069i9FC8018641B1D042/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_3-1756155301419.png" alt="CaitlinHalla_3-1756155301419.png" /></span></P><P>This configuration establishes our service identity, sets up the batch processor for efficient trace export, and configures the OTLP exporter to send traces directly to Splunk Observability Cloud.</P><P><STRONG>Note: </STRONG>in this current configuration, we are directly sending out telemetry data to our Splunk Observability Cloud backend. In most cases, you would want to set up an <A href="https://opentelemetry.io/docs/collector/" target="_blank">OpenTelemetry Collector</A> to collect, process, and export telemetry data to your observability backend. In a subsequent post, we’ll look at how we can update our current configuration to use the OpenTelemetry Collector for increased resilience, flexibility, and processing capabilities.</P><P><FONT size="5">Step 3: Initialize Instrumentation Components in Application</FONT></P><P><STRONG>Git commit:</STRONG> <A href="https://github.com/splunk/evangelism-public/commit/f7113fec" target="_blank">f7113fec</A></P><P>We next initialize the automatic instrumentations in our <FONT face="andale mono,times">application.ex</FONT> startup code:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_4-1756155301423.png" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40067i6E9C1D2019435AC9/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_4-1756155301423.png" alt="CaitlinHalla_4-1756155301423.png" /></span></P><P><STRONG>Notes on this setup:</STRONG></P><UL><LI>Cowboy Setup initializes the foundational HTTP server instrumentation</LI><LI>Phoenix Adapter setup ensures correct integration with Phoenix's HTTP server</LI><LI>Setup order matters. Initialize instrumentation libraries before your application starts accepting requests</LI><LI>Include logging to help verify configuration at startup</LI></UL><P>This setup initializes the dependencies referenced in the first commit in Step 1 and provides immediate visibility into HTTP requests, database queries, and GraphQL operations without any code changes. Every HTTP endpoint, database query, and resolver will automatically generate traces.</P><P><FONT size="5">Step 4: Add Custom Spans for Business Logic</FONT></P><P><STRONG>Git commit:</STRONG> <A href="https://github.com/splunk/evangelism-public/commit/9278ef62" target="_blank">9278ef62</A></P><P>While SDK-based automatic instrumentation covers the framework layers, we need to add custom spans to capture our application’s unique business logic. We do this by setting attributes on specific spans within our create function heads in our <FONT face="andale mono,times">resolvers.ex</FONT> file so that every time a spacewalk is created, custom telemetry data with specified attributes is exported to Splunk Observability Cloud:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_5-1756155301426.png" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40068i152CA7A4C86483B0/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_5-1756155301426.png" alt="CaitlinHalla_5-1756155301426.png" /></span></P><P><STRONG>Note:</STRONG> I’ve collapsed this method for brevity, but the full method and file can be viewed on GitHub in the <A href="https://github.com/splunk/evangelism-public/blob/main/manual-instrumentation/worms_in_space/lib/worms_in_space_web/api/resolvers/resolvers.ex#L46" target="_blank">manual-instrumentation/worms_in_space/lib/worms_in_space_web/api/resolvers/resolvers.ex</A> file.</P><P>These custom spans provide deep visibility into our spacewalk scheduling business logic, including success/failure rates, scheduling types, and performance metrics for critical operations. Some of the most common attributes to include in manual instrumentation are things like customer level or user ID and can be configured based on your unique business use-cases.</P><P><FONT size="5">Environment Configuration</FONT></P><P>Finally, we set our environment variables for deployment flexibility:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_6-1756155301428.png" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40070i897370D8F913FCFC/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_6-1756155301428.png" alt="CaitlinHalla_6-1756155301428.png" /></span></P><P>This configuration allows seamless deployment across development, staging, and production environments while maintaining complete trace context and correlation.</P><P><FONT size="6">What You'll See in Splunk Observability Cloud</FONT></P><P>With this instrumentation in place in just four commits, we immediately gain visibility into:</P><UL><LI>Complete request flows from HTTP request to database query with timing and status</LI><LI>GraphQL operation performance with query names and execution time&nbsp;&nbsp;</LI><LI>Database query analysis including SQL statements and execution time</LI><LI>Custom business metrics like the ones we added around spacewalks</LI><LI>Error tracking with full stack traces and contextual attributes</LI></UL><P>Going into Splunk Observability Cloud, we can visit the service overview of our Worms in Space backend in <A href="https://www.splunk.com/en_us/products/apm-application-performance-monitoring.html" target="_blank">Splunk Application Performance Monitoring</A>. Here we can see valuable service metrics, <A href="https://help.splunk.com/en/splunk-observability-cloud/monitor-application-performance/manage-services-spans-and-traces-in-splunk-apm/view-dependencies-among-your-services-in-the-service-map" target="_blank">Service Map</A>, <A href="https://help.splunk.com/en/splunk-observability-cloud/monitor-application-performance/analyze-services-with-span-tags-and-metricsets/analyze-service-performance-with-tag-spotlight" target="_blank">Tag Spotlight</A>, errors, and traces:&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_7-1756155301434.png" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40072i41F91D2FCFD506CF/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_7-1756155301434.png" alt="CaitlinHalla_7-1756155301434.png" /></span></P><P>If we open the traces from this view, we can see long running traces:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_8-1756155301440.png" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40071i34B0B8CE80509690/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_8-1756155301440.png" alt="CaitlinHalla_8-1756155301440.png" /></span></P><P>We can open one of those long running traces, and then we can select a span to view the custom attributes that we manually instrumented earlier:&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="CaitlinHalla_9-1756155301446.png" style="width: 999px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/40073iA93BB26515CADA98/image-size/large?v=v2&amp;px=999" role="button" title="CaitlinHalla_9-1756155301446.png" alt="CaitlinHalla_9-1756155301446.png" /></span></P><P>Manual instrumentation success <img class="lia-deferred-image lia-image-emoji" src="https://community.splunk.com/html/@65BA3226083A896B6028D25D1309E0A8/emoticons/1f389.png" alt=":party_popper:" title=":party_popper:" />.</P><P><FONT size="6">Wrap Up</FONT></P><P>In just four commits, we've transformed our un-instrumented Elixir backend into a fully observable service. The combination of automatic instrumentation for framework operations and manual instrumentation via custom spans for business logic provides complete visibility into application performance and behavior.</P><P>When issues arise at 2 AM, we’ll now know exactly which spacewalk scheduling operation failed, how long database queries are taking, and whether the problem is in our business logic or our infrastructure. This level of observability transforms mysterious production issues into clearly diagnosed problems with obvious solutions so space worms can seamlessly schedule their spacewalks at any time.</P><P>Join us in our next post to follow along, step-by-step, as we manually instrument the <SPAN>React-powered </SPAN>frontend of our Worms in Space <SPAN>application</SPAN>.&nbsp;</P><P>Ready to <SPAN>add context to</SPAN> your own backend<SPAN> apps</SPAN>? Start with <A href="https://opentelemetry.io/docs/languages/" target="_blank">OpenTelemetry's documentation on Language APIs and SDKs</A>, and use <A href="https://www.splunk.com/en_us/download/o11y-cloud-free-trial.html" target="_blank">Splunk Observability Cloud's 14-day free trial</A> to easily visualize your telemetry data in one unified backend platform.</P><P><FONT size="6">Resources</FONT></P><UL><LI><A href="https://community.splunk.com/t5/Community-Blog/Manual-Instrumentation-with-Splunk-Observability-Cloud-The-What/ba-p/751816" target="_blank">Manual Instrumentation with Splunk Observability Cloud: The What and Why</A></LI><LI><A href="https://help.splunk.com/en/splunk-observability-cloud/get-started/splunk-observability-cloud-overview/splunk-observability-cloud-overview" target="_blank">Splunk Observability Cloud Overview</A></LI><LI><A href="https://opentelemetry.io/docs/languages/" target="_blank">OpenTelemetry Language APIs &amp; SDKs</A></LI></UL><P>&nbsp;</P><P>&nbsp;</P><P>&nbsp;</P><P>&nbsp;</P> Thu, 28 Aug 2025 16:00:00 GMT CaitlinHalla 2025-08-28T16:00:00Z https://community.splunk.com/t5/Community-Blog/Manual-Instrumentation-with-Splunk-Observability-Cloud-How-to/ba-p/752186 <P>In this series, we're exploring manual instrumentation. We first looked at what manual instrumentation is and why you might use it.</P><P>In this post, we'll check out how we can implement manual instrumentation in the backend of our full-stack application. In just four commits, we'll take an un-instrumented code base from zero to full observability with OpenTelemetry and Splunk Observability Cloud.</P> Thu, 28 Aug 2025 16:00:00 GMT https://community.splunk.com/t5/Community-Blog/Manual-Instrumentation-with-Splunk-Observability-Cloud-How-to/ba-p/752186 CaitlinHalla 2025-08-28T16:00:00Z