Community Blog
Get the latest updates on the Splunk Community, including member experiences, product education, events, and more!

Hello POJO, meet nocode

jplumb
Splunk Employee
Splunk Employee

What's the Challenge?

Instrumentation agents are highly advanced bits of software that manipulate bytecode, intercept library calls, and wrap and observe other software in order to create telemetry. Their instrumentation modules are created specifically for common, popular frameworks and libraries, and the coverage for most use cases is astounding. Furthermore, we're seeing more and more libraries adopt OpenTelemetry as a native, first-class feature. However, off-the-shelf instrumentation may still be incomplete or may not provide exactly what a user wants.

With modern standards like OpenTelemetry, the first bit of guidance is to leverage manual instrumentation. In OTel, manual instrumentation allows software developers to insert annotations or wire up API calls in order to create the app-specific telemetry that they need. Because this manual instrumentation is created with code, it does increase the developer maintenance burden and, if not done with care, can clutter up the readability of core business logic.

Sometimes, manual instrumentation isn't possible or practical. While this should indeed be uncommon, it happens. This typically falls into one of three scenarios:

  • Third-party software whose source is unavailable or cannot be modified.
  • Organizational policy that prohibits observability teams from modifying source code.
  • Legacy code that is too risky or even impossible to rebuild or redeploy.

In these extreme cases, when auto-instrumentation doesn't suffice and manual instrumentation isn't possible, what can we do?

Declarative Instrumentation

The early generation of software observability vendors all offered their own unique way of allowing users to define instrumentation points declaratively. While slightly different in approach, they did more or less the same thing: they allowed instrumentation agents to create telemetry based on user-defined runtime configuration.

For example, AppDynamics offers a capability to instrument "plain ol' Java objects", often simply referred to as "POJO", with custom match rules. These rules are defined in the AppD Controller UI, which then passes this configuration down to the agents via its bespoke, proprietary protocol. The agent will then create AppD "Business Transactions" when these rules are matched.

Similarly, the bespoke New Relic agent has "XML Instrumentation". With this, users could write lots of angle brackets into a file in order to define how "Transactions" might be created. This file is then passed to the agent via configuration.

Naturally, Datadog created its own similar offering in the form of its dd.trace.methods configuration. Just like the other implementations, this allows users to define a set of classes and methods that should be instrumented, but in this case for tracing.

The OTel Java Way

The OpenTelemetry Java Agent also provides a flexible way to capture telemetry at the method level, via its "methods instrumentation". This is a first-class instrumentation that is automatically included in the OpenTelemetry distribution. Configuration of the methods instrumentation can be provided through environment variables and system properties, but because it can be quite verbose, using declarative config (yaml) is recommended.

While simple and useful, the instrumentation is quite limited. For instance, it doesn't provide a way to distinguish overloaded methods or customize the name of the new span. It also cannot surface method return values or parameter values.

Fortunately, the OTel Java Agent is open source, and pull requests that enhance the instrumentation capabilities are welcome.

The Splunk OTel Way

To provide additional user capabilities, the Splunk distribution of OpenTelemetry Java Instrumentation offers "nocode instrumentation". It provides a flexible way to declare instrumentation points by class and method, and also allows users to:

  • customize the span name
  • customize the span status
  • add custom span attributes

The span name, status, and attributes can be configured with JEXL expressions.

This feature is marked "under development" and is subject to breaking changes or removal. There is some effort to contribute this instrumentation to OpenTelemetry Java Instrumentation upstream, but this is very much still a work in progress.

Because JEXL evaluates arbitrary user expressions, it is both powerful and dangerous. Malicious use of JEXL in nocode instrumentation could, in the worst case, shut down the JVM or expose sensitive data.

Migrating from AppDynamics POJO rules

Users who are migrating from AppDynamics to Splunk Observability Cloud may wish to migrate their existing legacy POJO definitions. This can help to provide coverage in areas that are not readily handled by existing instrumentation in the Splunk OTel Java agent.

It's important to first acknowledge that these two observability platforms use very different data models. AppDynamics centers its bespoke data model on "Business Transactions" (BTs), while OpenTelemetry-based systems center on distributed traces made up of spans.

BTs and traces have a lot in common, but they are not the same thing!

We emphasize this because POJO rules are intended to trigger BT creation, while nocode instrumentation intends to create spans.

What's in a POJO Rule?

This dialog allows you to define a POJO rule in the AppDynamics controller:

pojo-rule-editor.png

It allows the user to choose what classes and methods should trigger the rule, and how those should be matched.

If you want to match a specific class and method, the mapping to nocode is straightforward. For example, a class named com.example.MyHotClass with a doSomething() method would appear like this in AppDynamics:

pojo-exact-class-method-rule.png

Its nocode YAML definition should be:

- class: com.example.MyHotClass
  method: doSomething
  span_kind: CLIENT
 

Note: It is highly recommended to provide the span_kind. This tells the Splunk Observability Cloud backend what kind of span this is.

Other Class Matches

Sometimes, POJOs are matched not purely on their exact class name, but on other class characteristics:

pojo-class-match-options.png

The nocode documentation covers these additional, more complicated cases, but we will provide examples here too.

Java class names should be fully qualified, including the package name.

Note: Regular expressions in YAML should usually be surrounded with single quotes. Periods in package names should be escaped with a backslash.

Matching by Interface

pojo-interface-match.png

You may have POJO rules that apply to all classes that implement an interface. When this is the case, use the super_type selector in your nocode definition. For example, to instrument all classes that implement the com.example.SelectionStrategy interface, use the following:

- class:
    super_type: com.example.SelectionStrategy
  method: choose
  span_kind: CLIENT
 

Note: It is not currently possible to perform non-exact matches with super_type class selectors in nocode

Matching by Superclass

Similar to the interface match above, some POJO rules might match classes based on their parent class or ancestry, using the "Super Class" criterion.

pojo-superclass-match.png

 

 

 

In nocode, this is also accomplished with the super_type matcher. For example, to match all subclasses of com.example.SuperServlet, you can use the following nocode YAML snippet:

- class:
    super_type: com.example.SuperServlet
  method: serve
  span_kind: SERVER
 

Note: It is not currently possible to perform non-exact matches with super_type class selectors in nocode

Matching by Annotation

 
 

pojo-annotation-match.png

 

AppDynamics POJO rules can select classes with a particular annotation. For example, you might target a class like this:

@MakeBizTransaction
class MyUsefulClass { 
  // ...
}
 

At the time of writing, nocode YAML expressions cannot match classes by annotation. This may be added in the future.

Note: Annotations must have a RUNTIME retention policy to be useful here.

Other Matching Operations

In addition to the precise "Equals" match, AppDynamics POJO definitions can use several other, more flexible matching predicates. This section describes those predicates and how to map them to nocode YAML definitions:

 

pojo-class-match-predicates.png

"Starts With"

class-match-starts-with.png

To perform a "Starts With" match in nocode, use a regular expression with the caret ('^') to match the start of the string, and use a wildcard through the end of the string. For example, to match any class that starts with the name com.example.Foo, the nocode equivalent should be:

- class:
    name_regex: '^com\.example\.Foo.*'
  method: bar
  span_kind: SERVER
 

This definition matches both com.example.FooBarImpl and com.example.FoodFight classes.

"Ends With"

class-match-ends-with.png

To perform an "Ends With" match in nocode, use a regular expression that starts with .*, contains the desired string, and ends with the $ end-of-string anchor. For example, to match any class that ends with the name common.Util, the nocode equivalent should be:

- class:
    name_regex: '.*common\.Util$'
  method: beep
  span_kind: SERVER
 

This definition matches both com.example.common.Util and com.example.uncommon.Util, but it would not match com.example.common.Utilities.

"Contains"

class-match-contains.png

A "Contains" match can be expressed with a regular expression that allows any characters before and after the desired string. For example, to match any class that contains the string FilterFactory, you could use the following nocode YAML expression:

- class:
    name_regex: '.*FilterFactory.*'
  method: filter
  span_kind: SERVER
 

This would match com.example.FilterFactory and com.example.AbstractFilterFactoryBaseImpl.

"Matches Reg Ex"

The AppD "Matches Reg Ex" is the same as the nocode name_regex.

Because over-matching with overly broad match expressions could generate undesired results, regular expressions should always be used with care.

"Is in List"

class-match-is-in-list.png

The "Is in List" matcher allows several class names to be matched. For example, you might want to match both com.example.Foo and com.example.Bar POJOs. To do this with nocode, use the or logic operator:

- class:
    or:
      - name: com.example.Foo
      - name: com.example.Bar
  method: bedazzle
  span_kind: SERVER
 

"Is Not Empty"

class-match-is-not-empty.png

I have no idea what this is, and I think it's unlikely that you have POJO rules that leverage this. If you do, please reach out and let me know.

Inverting the Method Name

The AppDynamics POJO rule creation dialog contains a gear icon that hides the NOT capability:

pojo-method-name-negation.png

Choosing this inverts the method-name match. In other words, it matches all methods on the selected classes that do not satisfy the selection criteria. In nocode, use the not operator to do the same. For example, to match all methods whose name is not swizzle:

- class: com.example.MyClass
  method: 
    not:
      name: swizzle
  span_kind: SERVER
 

What About Splitting?

The "Add Rule" screen also lets AppDynamics users split Business Transactions.

pojo-business-transaction-splitting.png

AppDynamics Business Transactions have no direct equivalent in the OpenTelemetry data model, which uses traces and spans. As a result, nocode cannot split a Business Transaction. A span created by nocode will either be the root span of a new trace or be created within the existing trace context.

nocode for existing spans

To add attributes to the current span instead of creating a new span, set current_span: true in the nocode rule. See the documentation for additional details about this feature.

Conclusion

Sometimes, you just have to do it the hard way. Fortunately, there are options in the form of declarative instrumentation to help in these situations.

The Splunk Distribution of OpenTelemetry Java Instrumentation provides a powerful nocode instrumentation module that can be configured to cover most of the common use cases from the AppDynamics POJO rule definitions. This allows OpenTelemetry spans to be created for user code or third-party libraries for which there is no existing instrumentation.

The mappings described in this article can help users who are migrating from AppDynamics to Splunk Observability Cloud and have existing gaps that need instrumentation coverage for continuity. A word of caution: Declarative instrumentation, like nocode, should usually be used as a last resort after determining that other instrumentation doesn't exist or that manual instrumentation cannot be added.

Use care when defining these rules: an improperly written rule could negatively affect an application. In the worst case, evaluating an arbitrary Java expression could cause the application to terminate unexpectedly or degrade performance. A misconfigured rule could also generate unwanted volumes of telemetry.

Contributors

Write for the Community Blog

Have a tip, real-world lesson, or project to share? We’ll partner with you to refine your draft, sharpen your headings, and spotlight your work.

Get Updates on the Splunk Community!

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...

Move CrowdStrike FDR ingestion to Data Inputs with a guided migration

CrowdStrike Falcon Data Replicator data is essential for security investigations, detections, and response. It ...

Make Your Machine Data AI-Ready in Splunk

Every team is getting ready for AI agents.    Watch On-Demand   In this Tech Talk, Splunk platform specialists ...