Instrumentation is usually the last step or even an afterthought when building out a project. The feature ships, the sprint closes, and the OpenTelemetry work lands in the backlog until something breaks in production and everyone suddenly needs the telemetry that was never added. Setting up SDKs, custom spans, and metrics by hand takes real time, which puts it in direct competition with feature work.
Splunk Observability Studio levels the playing field and moves this instrumentation work into the IDE, where an AI coding agent can implement most of it at code creation time. It takes visibility even further than flipping on auto-instrumentation. It audits what your service should be emitting, adds the custom spans and metrics auto-instrumentation can't know about, and then verifies that the telemetry exists. It's free, open source, and it includes a local telemetry workspace so results are visible immediately.
In this post, we'll walk through what's in the project today and how to start using it.
What’s included
Splunk Observability Studio is made up of two main pieces that work together:
- Agent skills give your AI coding agent (Codex, Claude Code, Cursor, or Kiro) repeatable workflows for auditing, instrumenting, and verifying OpenTelemetry instrumentation in a repo.
- A local OpenTelemetry workspace: a collector with OTLP receivers, a Telemetry Explorer UI, a REST API, and an MCP server, all running on localhost. A companion editor extension for Visual Studio Code, Kiro, and Cursor can bring this Telemetry Explorer directly into your editor.
All of the instrumentation it creates is standard OpenTelemetry, so it’s vendor neutral and isn't tied to any particular backend observability platform.
How to use it
The primary workflow involves three core skills you run from your coding agent, each feeding the next. Audit finds the gaps. Instrument closes them. Verify proves they're closed. You stay in control at every step and nothing changes in your code that you don't explicitly select. Let’s get into each.
Note: this post refers to the skills by name – otel-audit, otel-instrument, otel-verify. Formally, Claude Code, Cursor, and Kiro surface them as slash commands (/otel-audit), while Codex uses $otel-audit syntax. But the skills are forgiving so typing either form or even skipping the syntax entirely and asking in plain language, e.g. "Audit this service for observability gaps," will take you through the same workflow.
Step 1: audit observability coverage
The otel-audit skill scans for observability coverage gaps without modifying any code. Use it on greenfield projects or existing applications to quickly find missing SDK setup, routes with no telemetry, operations that should include business metrics but don't, spans that start in one place but are never propagated.
The audit result is a machine-readable report plus an interactive HTML report where each finding can be reviewed and acted upon:
Even if you never let the agent change a line of code, this audit report alone is a huge win. It’s a prioritized, actionable instrumentation to-do list.
Step 2: instrument and verify
After selecting and saving findings, the report hands back a generated $otel-instrument command that can apply your selections to your codebase. Running it adds the SDK setup, auto-instrumentation, and targeted custom spans and metrics, then writes a second interactive report that maps each selected gap to the code changes made and the telemetry you should now see:
otel-instrument also runs the otel-verify workflow by default, which checks the instrumentation with application code tests and real OTLP data so gaps are confirmed closed rather than assumed closed.
This verification step means you’re not handing instrumentation off to an agent and hoping that it instrumented everything correctly. You get a real report that maps each gap to a diff and to the telemetry data that’s actually flowing in.
Step 3: view telemetry locally
Running obstudio starts OTLP receivers and the Telemetry Explorer. Run the app, and the newly instrumented spans, metrics, and logs show up in real time. The Metrics tab charts custom metrics as they arrive.
Here we can see checkout.request.duration from a demo checkout service:
Traces render as a waterfall, including traces that cross services. Here we have a checkout request traced across two services, with the database query, the downstream payment call, and an error visible span by span:
Logs land here, as well, so we can see logs with severity and service context right alongside traces and metrics:
The Telemetry Explorer is not a replacement for an observability backend. It answers a development-time question: is my service actually emitting the spans, metrics, and dimensions I think it is? Normally answering that means deploying and waiting. Now it’s a glance at a tab in your editor and can be answered before the code even leaves the laptop.
Step 4: send it to Splunk Observability Cloud
If you use Splunk Observability Cloud, Observability Studio connects the new instrumentation to it in a few ways.
Data can be forwarded from the dev loop by setting a realm and access token in Observability Studio's env file. This forwards local metrics and traces to Splunk Observability Cloud, where the service shows up in APM like any other.
Need a Splunk Observability Cloud Free Edition? Your IDE also has you covered for that:
Detectors and dashboards can also be configured as code. The splunk-configure skill reads the audit report and generates ready-to-apply Terraform for Splunk Observability Cloud detectors based on the newly added telemetry. A new metric isn't worth much if nothing charts it or alerts on it, so content creation is part of the workflow.
Finally, you can choose to only publish the observability gaps using the splunk-detector-publish and splunk-dashboard-publish skills, which diff the generated Terraform against what already exists in your Splunk org and create only what's missing. You can even preview the generated dashboards in the Telemetry Explorer’s Dashboards tab plotted against your local telemetry data before ever running terraform apply:
If you've read our Observability as Code posts, this is that same practice, but instead generated from an audit of your actual application.
Getting started
Everything lives in the open source repo at github.com/signalfx/obstudio. Download the latest release for your platform and install the skills for your agent:
obstudio install --target=codex,claude-code,cursor,kiro To work from your editor, install the free extension from the Visual Studio Marketplace or Open VSX for Kiro and Cursor:
You can then open the Telemetry Explorer with Splunk Observability Studio: Open Observer from the Command Palette.
Wrap up
Splunk Observability Studio makes visibility easier than ever. No more blind spots shipping into production, and no more instrumentation debt waiting for an incident to turn it into a priority. Audit an existing service to find what it isn’t telling you, instrument new code as you write it, and verify it’s all real before anything ships – all from within your IDE.
Ready to try it on one of your own services? Clone the repo, run the otel-audit command, and see what it finds. Need a backend to send all that telemetry to? Start with Splunk Observability Cloud Free Edition to get enterprise-grade observability with no credit card required.
Resources
Write for the Community Blog
Have a tip, real-world lesson, or project to share? We’ll partner with you to refine your draft, sharpen your headings, and spotlight your work.
