This article is a follow-up to my previous article posted on the OpenTelemetry Blog, "Your Critical Legacy App is a Black Box? Let's Change That in 5 Minutes!" . In this sequel, I’ll demonstrate how you can achieve the same (and even more) with the Splunk OpenTelemetry Collector and Splunk Cloud. Additionally, we’ll cover log collection, an important aspect that was not addressed in the original article.
OpenTelemetry for Legacy Apps? Yes, You Can!
Do you have a legacy app in your organization that is crucial but completely mysterious? Maybe you’d like to ask the developers for more details but… the entire team who built it 20 years ago has long since moved on. You’re not alone! Many organizations depend on reliable systems that offer little visibility into what’s happening beneath the surface. While migrating or rewriting these applications might be the long-term goal, you need actionable insights today.
OpenTelemetry. Does it ring a bell? Maybe you’ve heard of it and immediately think of modern, cloud-native applications. That’s definitely true – OpenTelemetry delivers enormous value for loosely coupled, distributed systems by enabling powerful observability and monitoring.
But its benefits don’t end there. OpenTelemetry can also help shine a light on your legacy applications by collecting essential metrics and logs – even without touching the original code. This means you can monitor application performance, troubleshoot issues, and make informed decisions about modernization, all while reducing risk.
Shedding Light on Previously Unseen Systems
To mimic a real-world integration scenario, we’ve developed a sample application that demonstrates how legacy systems can interact with modern platforms. In this setup, a core executable written in C (representing a traditional system-level process) launches a Java Virtual Machine (JVM) to delegate specific tasks (such as processing a transaction or handling a data record) to Java code. While your organization’s exact application stack may differ, this pattern is common in environments where established C-based infrastructure needs to leverage the flexibility and modern features of Java. Typical use cases include embedding updated business logic within legacy workflows, bridging data processing between old and new systems, or enabling advanced scripting capabilities. The flow for our application is following:
C Application (legacy_app) → starts JVM → invokes Java method to process a task
Before moving forward, let’s double-check that our application is up and running.
./legacy_app
[C Wrapper] System online. Began monitoring assembly line...
Press Ctrl+C to stop.
[C Wrapper] Reading new Part ID from assembly line: 1001
Sep 20, 2025 10:25:07 PM LegacyJavaProcessor processData
INFO: [Java Processor] Received Part ID 1,001. Fetching processing parameters...
Sep 20, 2025 10:25:07 PM LegacyJavaProcessor processData
INFO: [Java Processor] Part ID 1,001 processed successfully.
[C Wrapper] Processing request for Part ID 1001 completed successfully.
[C Wrapper] Reading new Part ID from assembly line: 1002
Sep 20, 2025 10:25:08 PM LegacyJavaProcessor processData
INFO: [Java Processor] Received Part ID 1,002. Fetching processing parameters...
Sep 20, 2025 10:25:08 PM LegacyJavaProcessor processData
INFO: [Java Processor] Part ID 1,002 processed successfully.
[C Wrapper] Processing request for Part ID 1002 completed successfully.
While our application runs as expected, we lack any real insight into its internal performance or health. This means we are unable to address crucial questions: Is the system able to handle peak business loads? Are there hidden memory issues that could lead to crashes during critical processing windows? Could a slowdown in this application be the underlying reason for recent production line stops? Without proper visibility, it’s difficult to proactively identify and resolve issues before they impact users or business operations.
Everything changes when you run into your company’s Splunk Admin in your company cafeteria, who introduces you to OpenTelemetry. She explains how easily you can instrument your application to send metrics and logs (we will need traces too but we’ll clarify that later in the article) directly to your organization’s Splunk Cloud instance. The solution is surprisingly straightforward – just three simple steps stand between your legacy application and deep, actionable insights into its performance and business KPIs.
Depending on the programming language and framework, you may need to modify your code to add instrumentation, or you can leverage zero-code instrumentation for a much simpler setup. In our case, the core of our application is written in Java (even though part of it is implemented in C), which allows us to take full advantage of OpenTelemetry’s zero-code instrumentation. You can quickly enable automated data collection and gain real-time insights into your application’s behavior – helping you identify system load, memory usage, and performance bottlenecks before they impact your users or business. It really is as simple as 1, 2, 3:
Step 1: Install the Splunk OpenTelemetry Collector
Step 2: Download the Splunk Distribution of OpenTelemetry Java
Step 3: Configure a few required environment variables
With just these steps, you open the door to comprehensive observability for your legacy systems, transforming guesswork into data-driven decisions. Let us know go through all these steps in more details.
Step 1: Install the Splunk OpenTelemetry Collector
There are several ways to deploy the Splunk OpenTelemetry Collector, such as running it in a Docker container or installing it directly on a Linux or Windows host. However, for our purposes, we’ll focus primarily on the Collector’s configuration rather than the specific deployment method. Below is a minimal OpenTelemetry Collector configuration to help you get started:
# otelcol-config.yaml
receivers:
# OTLP receiver: This is where your Java application will send its traces, metrics, and logs.
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
# Batch processor: Improves efficiency by sending data in batches.
batch:
send_batch_size: 10000
timeout: 10s
exporters:
# Splunk HEC exporter: Sends all telemetry data (logs, metrics, traces) to Splunk Cloud Platform.
# The exporter will convert OTLP metrics and traces into HEC-compatible events.
splunk_hec/logs:
endpoint: "XXXXXXXXXX"
token: "XXXXXXXXXX"
source: "otel-collector"
sourcetype: "otel:logs"
index: "main"
splunk_hec/traces:
endpoint: "XXXXXXXXXX"
token: "XXXXXXXXXX"
source: "otel-collector"
sourcetype: "otel:traces"
index: "main"
splunk_hec/metrics:
endpoint: "XXXXXXXXXX"
token: "XXXXXXXXXX"
source: "otel-collector"
sourcetype: "otel:metrics"
index: "metrics"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [splunk_hec/traces]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [splunk_hec/metrics]
logs:
receivers: [otlp]
processors: [batch]
exporters: [splunk_hec/logs]
There are two important points to clarify at this stage:
Why do we need traces?
As you’ll see later, we’ll be leveraging the otel.instrumentation.methods.include configuration property. Even if you can’t modify the application code, you can configure the Java agent to automatically capture spans around specific methods. This allows you to derive valuable APM metrics such as Calls per Minute, Average Response Time per Minute, or Errors per Minute – providing much deeper insight into your application’s behavior and performance.
Why are we sending metrics, logs, and traces to Splunk Cloud instead of Splunk Observability Cloud?
That’s a great question! Splunk Observability Cloud is a powerful option for advanced real-time observability, especially in cloud-native and microservices environments. However, in our case, we assume that Splunk Cloud is already available in the organization. Splunk Cloud still offers robust features for log and event management, search, alerting, and reporting. For many legacy applications, these capabilities provide sufficient visibility and insights, even without some of the advanced tracing and metrics correlation features found in Observability Cloud. This makes Splunk Cloud a practical and effective choice for our needs.
Step 2: Download Splunk Distribution of OpenTelemetry Java
This step is simple and quick. Just head over to Splunk’s GitHub and download the latest version of the Splunk Distribution of OpenTelemetry Java agent. The releases page provides the most up-to-date agent JAR file, ready to be attached to your Java application.
You might be wondering why you should choose the Splunk Distribution over the upstream OpenTelemetry Java instrumentation. The Splunk Distribution of OpenTelemetry Java enhances standard OpenTelemetry instrumentation with powerful additions, including more comprehensive JVM and application metrics collection, AlwaysOn Profiling, and seamless integration with Splunk Observability Cloud. It also offers out-of-the-box exporter configurations, trace metadata injection into logs, and frequent updates with Splunk support, ensuring faster access to new features and fixes. This makes it an ideal choice for organizations using Splunk’s platform, delivering deeper Java application observability with minimal performance impact.
To get more details please visit Splunk Documentation Page for Java Agent
Step 3: Configure a few required environment variables
Now it’s time for the final step. In your terminal, you’ll configure the agent by setting a few environment variables - no need to modify the application code itself! This approach makes the setup process quick and non-intrusive. With just a couple of simple commands, you’ll enable the agent to collect and send metrics, logs and traces to OpenTelemetry collector.
As mentioned earlier in the article, for scenarios where you can’t modify the application’s source code, the OpenTelemetry Java agent provides a powerful feature: otel.instrumentation.methods.include. With this setting, you can instruct the agent to automatically create spans around specific methods of interest. For instance, in our case we looked into the code and identified LegacyJavaProcessor.processData as a critical method, so we’ll configure OTEL_INSTRUMENTATION_METHODS_INCLUDE environment variable accordingly to capture spans:
# 1. Give our service a descriptive name
export OTEL_SERVICE_NAME=legacy-part-processor
# 2. Enable metrics, logs and traces exporters
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
# 3. Enable runtime metrics for Java 17
export OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_JAVA17_ENABLED=true
# 4. Tell the agent which method to instrument
export OTEL_INSTRUMENTATION_METHODS_INCLUDE="LegacyJavaProcessor[processData]"
# 5. Attach the Splunk Distribution of OpenTelemetry Java agent
export _JAVA_OPTIONS="-javaagent:./ splunk-otel-javaagent.jar"
Let’s go ahead and start the application to confirm that the Splunk Distribution of OpenTelemetry Java’s Java Agent is properly attached and running as expected:
./legacy_app
Picked up _JAVA_OPTIONS: -javaagent:./splunk-otel-javaagent.jar
[otel.javaagent 2025-09-20 23:01:49:331 -0400] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: splunk-2.20.0-otel-2.20.0
[C Wrapper] System online. Began monitoring assembly line...
Press Ctrl+C to stop.
[C Wrapper] Reading new Part ID from assembly line: 1001
Sep 20, 2025 11:01:55 PM LegacyJavaProcessor processData
INFO: [Java Processor] Received Part ID 1,001. Fetching processing parameters...
Sep 20, 2025 11:01:55 PM LegacyJavaProcessor processData
INFO: [Java Processor] Part ID 1,001 processed successfully.
[C Wrapper] Processing request for Part ID 1001 completed successfully.
[C Wrapper] Reading new Part ID from assembly line: 1002
Sep 20, 2025 11:01:57 PM LegacyJavaProcessor processData
INFO: [Java Processor] Received Part ID 1,002. Fetching processing parameters...
Sep 20, 2025 11:01:57 PM LegacyJavaProcessor processData
INFO: [Java Processor] Part ID 1,002 processed successfully.
[C Wrapper] Processing request for Part ID 1002 completed successfully.
Turning Telemetry Into Actionable Dashboards
With everything set up, the advantages of this approach become clear right away. The Splunk OpenTelemetry Collector is now exporting your metrics, logs, and traces directly to Splunk Cloud using the HTTP Event Collector (HEC). By heading to Splunk Cloud, you can use just a few simple queries to quickly build your first dashboard, instantly gaining visibility into key metrics like average CPU utilization, average memory consumption, and time spent in garbage collection which are basic application health KPIs.
The trace spans you are capturing provide the raw material for even richer dashboards. In Splunk Cloud, you can easily add charts for three crucial performance metrics: calls per minute, average response time, and errors per minute.
Additionally, you can leverage the collected logs to extract operational metrics, such as tracking how many parts were processed successfully – giving you end-to-end observability and actionable insights into your application’s health and performance.
Presented below is a sample dashboard in Splunk, demonstrating how powerful insights can be derived from just a couple of SPL queries. It leverages comprehensive metrics, logs, and traces, all collected via the OpenTelemetry Collector, to highlight key performance indicators (KPIs) for our legacy application that cover both basic application health and actual operational performance.
Key Takeaways & Next Steps
Stop flying blind. Even partial visibility into your legacy systems is a massive improvement! With minimal effort you can instrument your "black box" applications, gaining critical insights without disruptive code changes or expensive overhauls. This isn't just a technical win; it's a strategic advantage, enabling data-driven decisions, proactive problem-solving, and building confidence for future planning.
Your legacy application might seem resistant to modern tooling, but it's always crucial to verify your tech stack. The OpenTelemetry ecosystem is rapidly advancing, offering agent-based or sidecar instrumentation that often requires zero code modification. This makes extracting valuable telemetry from even the oldest systems more accessible than ever before. So go ahead, blow the dust off that old server – OpenTelemetry is ready to help it shine!
