Instrumentation agents are highly advanced bits of software that manipulate bytecode, intercept library calls, and wrap and observe other software in order to create telemetry. Their instrumentation modules are created specifically for common, popular frameworks and libraries, and the coverage for most use cases is astounding. Furthermore, we're seeing more and more libraries adopt OpenTelemetry as a native, first-class feature. However, off-the-shelf instrumentation may still be incomplete or may not provide exactly what a user wants.
With modern standards like OpenTelemetry, the first bit of guidance is to leverage manual instrumentation. In OTel, manual instrumentation allows software developers to insert annotations or wire up API calls in order to create the app-specific telemetry that they need. Because this manual instrumentation is created with code, it does increase the developer maintenance burden and, if not done with care, can clutter up the readability of core business logic.
Sometimes, manual instrumentation isn't possible or practical. While this should indeed be uncommon, it happens. This typically falls into one of three scenarios:
In these extreme cases, when auto-instrumentation doesn't suffice and manual instrumentation isn't possible, what can we do?
The early generation of software observability vendors all offered their own unique way of allowing users to define instrumentation points declaratively. While slightly different in approach, they did more or less the same thing: they allowed instrumentation agents to create telemetry based on user-defined runtime configuration.
For example, AppDynamics offers a capability to instrument "plain ol' Java objects", often simply referred to as "POJO", with custom match rules. These rules are defined in the AppD Controller UI, which then passes this configuration down to the agents via its bespoke, proprietary protocol. The agent will then create AppD "Business Transactions" when these rules are matched.
Similarly, the bespoke New Relic agent has "XML Instrumentation". With this, users could write lots of angle brackets into a file in order to define how "Transactions" might be created. This file is then passed to the agent via configuration.
Naturally, Datadog created its own similar offering in the form of its dd.trace.methods configuration. Just like the other implementations, this allows users to define a set of classes and methods that should be instrumented, but in this case for tracing.
The OpenTelemetry Java Agent also provides a flexible way to capture telemetry at the method level, via its "methods instrumentation". This is a first-class instrumentation that is automatically included in the OpenTelemetry distribution. Configuration of the methods instrumentation can be provided through environment variables and system properties, but because it can be quite verbose, using declarative config (yaml) is recommended.
While simple and useful, the instrumentation is quite limited. For instance, it doesn't provide a way to distinguish overloaded methods or customize the name of the new span. It also cannot surface method return values or parameter values.
Fortunately, the OTel Java Agent is open source, and pull requests that enhance the instrumentation capabilities are welcome.
To provide additional user capabilities, the Splunk distribution of OpenTelemetry Java Instrumentation offers "nocode instrumentation". It provides a flexible way to declare instrumentation points by class and method, and also allows users to:
The span name, status, and attributes can be configured with JEXL expressions.
This feature is marked "under development" and is subject to breaking changes or removal. There is some effort to contribute this instrumentation to OpenTelemetry Java Instrumentation upstream, but this is very much still a work in progress.
Because JEXL evaluates arbitrary user expressions, it is both powerful and dangerous. Malicious use of JEXL in nocode instrumentation could, in the worst case, shut down the JVM or expose sensitive data.
Users who are migrating from AppDynamics to Splunk Observability Cloud may wish to migrate their existing legacy POJO definitions. This can help to provide coverage in areas that are not readily handled by existing instrumentation in the Splunk OTel Java agent.
It's important to first acknowledge that these two observability platforms use very different data models. AppDynamics centers its bespoke data model on "Business Transactions" (BTs), while OpenTelemetry-based systems center on distributed traces made up of spans.
BTs and traces have a lot in common, but they are not the same thing!
We emphasize this because POJO rules are intended to trigger BT creation, while nocode instrumentation intends to create spans.
This dialog allows you to define a POJO rule in the AppDynamics controller:
It allows the user to choose what classes and methods should trigger the rule, and how those should be matched.
If you want to match a specific class and method, the mapping to nocode is straightforward. For example, a class named com.example.MyHotClass with a doSomething() method would appear like this in AppDynamics:
Its nocode YAML definition should be:
- class: com.example.MyHotClass
method: doSomething
span_kind: CLIENT
Note: It is highly recommended to provide the span_kind. This tells the Splunk Observability Cloud backend what kind of span this is.
Sometimes, POJOs are matched not purely on their exact class name, but on other class characteristics:
The nocode documentation covers these additional, more complicated cases, but we will provide examples here too.
Java class names should be fully qualified, including the package name.
Note: Regular expressions in YAML should usually be surrounded with single quotes. Periods in package names should be escaped with a backslash.
You may have POJO rules that apply to all classes that implement an interface. When this is the case, use the super_type selector in your nocode definition. For example, to instrument all classes that implement the com.example.SelectionStrategy interface, use the following:
- class:
super_type: com.example.SelectionStrategy
method: choose
span_kind: CLIENT
Note: It is not currently possible to perform non-exact matches with
super_typeclass selectors innocode
Similar to the interface match above, some POJO rules might match classes based on their parent class or ancestry, using the "Super Class" criterion.
In nocode, this is also accomplished with the super_type matcher. For example, to match all subclasses of com.example.SuperServlet, you can use the following nocode YAML snippet:
- class:
super_type: com.example.SuperServlet
method: serve
span_kind: SERVER
Note: It is not currently possible to perform non-exact matches with
super_typeclass selectors innocode
AppDynamics POJO rules can select classes with a particular annotation. For example, you might target a class like this:
@MakeBizTransaction
class MyUsefulClass {
// ...
}
At the time of writing, nocode YAML expressions cannot match classes by annotation. This may be added in the future.
Note: Annotations must have a RUNTIME retention policy to be useful here.
In addition to the precise "Equals" match, AppDynamics POJO definitions can use several other, more flexible matching predicates. This section describes those predicates and how to map them to nocode YAML definitions:
"Starts With"
To perform a "Starts With" match in nocode, use a regular expression with the caret ('^') to match the start of the string, and use a wildcard through the end of the string. For example, to match any class that starts with the name com.example.Foo, the nocode equivalent should be:
- class:
name_regex: '^com\.example\.Foo.*'
method: bar
span_kind: SERVER
This definition matches both com.example.FooBarImpl and com.example.FoodFight classes.
To perform an "Ends With" match in nocode, use a regular expression that starts with .*, contains the desired string, and ends with the $ end-of-string anchor. For example, to match any class that ends with the name common.Util, the nocode equivalent should be:
- class:
name_regex: '.*common\.Util$'
method: beep
span_kind: SERVER
This definition matches both com.example.common.Util and com.example.uncommon.Util, but it would not match com.example.common.Utilities.
A "Contains" match can be expressed with a regular expression that allows any characters before and after the desired string. For example, to match any class that contains the string FilterFactory, you could use the following nocode YAML expression:
- class:
name_regex: '.*FilterFactory.*'
method: filter
span_kind: SERVER
This would match com.example.FilterFactory and com.example.AbstractFilterFactoryBaseImpl.
The AppD "Matches Reg Ex" is the same as the nocode name_regex.
Because over-matching with overly broad match expressions could generate undesired results, regular expressions should always be used with care.
The "Is in List" matcher allows several class names to be matched. For example, you might want to match both com.example.Foo and com.example.Bar POJOs. To do this with nocode, use the or logic operator:
- class:
or:
- name: com.example.Foo
- name: com.example.Bar
method: bedazzle
span_kind: SERVER
I have no idea what this is, and I think it's unlikely that you have POJO rules that leverage this. If you do, please reach out and let me know.
The AppDynamics POJO rule creation dialog contains a gear icon that hides the NOT capability:
Choosing this inverts the method-name match. In other words, it matches all methods on the selected classes that do not satisfy the selection criteria. In nocode, use the not operator to do the same. For example, to match all methods whose name is not swizzle:
- class: com.example.MyClass
method:
not:
name: swizzle
span_kind: SERVER
The "Add Rule" screen also lets AppDynamics users split Business Transactions.
AppDynamics Business Transactions have no direct equivalent in the OpenTelemetry data model, which uses traces and spans. As a result, nocode cannot split a Business Transaction. A span created by nocode will either be the root span of a new trace or be created within the existing trace context.
To add attributes to the current span instead of creating a new span, set current_span: true in the nocode rule. See the documentation for additional details about this feature.
Sometimes, you just have to do it the hard way. Fortunately, there are options in the form of declarative instrumentation to help in these situations.
The Splunk Distribution of OpenTelemetry Java Instrumentation provides a powerful nocode instrumentation module that can be configured to cover most of the common use cases from the AppDynamics POJO rule definitions. This allows OpenTelemetry spans to be created for user code or third-party libraries for which there is no existing instrumentation.
The mappings described in this article can help users who are migrating from AppDynamics to Splunk Observability Cloud and have existing gaps that need instrumentation coverage for continuity. A word of caution: Declarative instrumentation, like nocode, should usually be used as a last resort after determining that other instrumentation doesn't exist or that manual instrumentation cannot be added.
Use care when defining these rules: an improperly written rule could negatively affect an application. In the worst case, evaluating an arbitrary Java expression could cause the application to terminate unexpectedly or degrade performance. A misconfigured rule could also generate unwanted volumes of telemetry.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Have a tip, real-world lesson, or project to share? We’ll partner with you to refine your draft, sharpen your headings, and spotlight your work.