Community Blog
Get the latest updates on the Splunk Community, including member experiences, product education, events, and more!

Five Subtly Different Ways of Adding Manual Instrumentation in Java

Splunk Employee
Splunk Employee

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in touch!

Please make sure to check out the code and install Java 11, at minimum, to execute this example.


./gradlew run

Note: Java instrumentation debug logging is enabled, so it's fairly verbose.


At the top level, we get a nice trace consisting of 5 spans:



Let's dive a little deeper into each one:

Parent Method (root span)

We see that the root span attained its name from the simple use of the @WithSpan attribute. The operation name is SpanAttributesMain.parentMethod which is automatically derived from the name of the class and the name of the method being instrumented.

It is worth pointing out that, by default, method parameters are NOT included as attributes. In this example, you may note the absence of counter anywhere on the span.

In the list of tags (OpenTelemtry attributes) you will notice code.function and code.namespace, which are standard OpenTelemetry semantic conventions for source code. This will be true of all other spans that used @WithSpan below.



Method 2

Method 2 looks a lot like the root span, with a few exceptions. First, the name has been customized so that instead of the raw class and method name, we see Method Two. This is customized by passing a string value to the @WithSpan annotation.

Additionally, there is now a new custom attribute on the span: bucket. The @SpanAttribute parameter annotation is used to automatically append a parameter as a tag to a span. NOTE: The automatic detection of parameter names is only available through java reflection when the -parameters argument is passed to the compiler (see the build.gradle.kts to see how this was accomplished). Without -parameters passed to javac, your @SpanAttribute argument will NOT be on the span unless you give it a name (see method 3 below).



Method 3

Method 3 is similar to Method Two but it customizes the attribute name. Instead of bucket, it has specified custom.example.bucket.value by passing an argument to the @SpanAttribute annotation.



Method 4

Method 4 falls back to the generic @WithSpan naming but has manually added an attribute to the currently scoped span with code.

The code to do this looks like this:

Span.current().setAttribute("special.bucket.value", bucket);


Method 5

Span 5 is created entirely with manual instrumentation. The span name (method 5) and custom attribute name (method5.bucket.value) are both generated with java code manually written into the method.

This approach is the most flexible, but requires more effort. It's worth noting that the instrumentation tags like code.function, code.namespace,, and otel.library.version are not added automatically here.



Metricizing Span Attributes

From Tag Spotlight, you can click "New MetricSet" to define a new metric set for our custom attribute:



Now that we've created some span data with our custom bucket attribute, we can create metrics for it.



As you can see, we can compare times for each bucket (which in our contrived example is not exciting) or graph them to do additional analysis.

—Jason Plumb, Principal Software Engineer, OpenTelemetry Java Contributor

Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...