You open a trace in Splunk Observability Cloud and everything looks fine. One root span, order-pipeline , with four children underneath it: fetch , validate , transform , persist . You can see how long each one took, right there in the waterfall. But you want more than one trace. You want to know: is transform getting slower over the last week? Is validate throwing more errors than usual? You go looking for that chart, and it doesn't exist. No RED metrics, no MMS, nothing to alert on. Just a trace you'd have to open by hand, over and over, to notice a trend. That gap isn't a bug. It's how APM platforms are built, and there's a clean way to close it without touching your application code. Why internal spans go dark, and why relabeling them isn't the fix   Splunk Observability Cloud, like most APM platforms, derives its RED metrics (rate, errors, duration) and MMS's from service-boundary spans only — SERVER and CLIENT . Those are the spans that represent a request crossing into or out of a service, so they're the natural unit for "how is this service doing." Everything nested underneath, every INTERNAL span, is treated as detail inside that request. Useful when you're looking at one trace. Invisible everywhere else. The full trace: one root span, four internal spans nested underneath it, all fully instrumented.   So fetch , validate , transform , and persist sit there, fully instrumented, fully visible in the trace view, and completely absent from any dashboard. No per-operation rate. No per-operation error rate. No detector you can point at transform specifically.   The default dashboard for this service: one endpoint, the root span. The four internal spans underneath it aren't in the picture at all.   The instinct at this point is to just fix it in code: change the SpanKind of those four spans from INTERNAL to SERVER . Don't do this. SpanKind.SERVER is defined by the OpenTelemetry spec for the server side of an inbound RPC or similar remote request — not for a sequential step inside a single process. Relabeling transform as SERVER doesn't just bend a naming convention. It misrepresents the actual call structure of your trace, and any tooling that reasons about span kind — service maps, dependency graphs, anything downstream — will draw the wrong conclusions about what's talking to what. You'd be trading one visibility problem for a subtler, harder-to-spot one. The idiomatic fix: span_metrics, forked off with routing OpenTelemetry already has an answer for this, and it doesn't involve touching a single line of instrumentation. It's a Collector component called span_metrics — a connector purpose-built to turn spans into metrics. Not a workaround. The spec-sanctioned way to get RED metrics out of any span, INTERNAL included.A connector, in OTel Collector terms, is a component that acts as an exporter for one pipeline and a receiver for another. That's what lets you bolt derived signals onto an existing trace flow, declaratively, entirely in config. No code, no redeploys of the app. Two connectors do the work here: routing  forks the trace stream. Configured with action: copy rather than a plain route, it sends a matching span to *both* its normal destination *and* a second pipeline — not one or the other. That distinction matters: a plain route would divert your traces away from Splunk Observability Cloud the moment you turned this on. copy keeps everything flowing exactly as before, and adds a second path alongside it. span_metrics  receives that second stream and aggregates it into RED-style metrics — count, duration, errors — grouped by span name, flushed on a fixed interval. Those metrics then flow into your existing metrics pipeline like any other metric source. Here's the shape of it: Diagram of the OTel Collector pipelines: otlp receiver into the routing connector, forking into the traces/default pipeline and the traces/route-to-spanmetrics pipeline, with the latter feeding the span_metrics connector into the metrics pipeline. Traces still land in Splunk Observability Cloud exactly as they did before. The only thing that's new is the second path, quietly generating metrics from the same data. What you get, concretely   Once this is wired up, span_metrics emits two metrics: traces.span.metrics.calls and traces.span.metrics.duration . In Splunk Observability Cloud, that's enough to build request rate, latency percentiles, and error rate — both overall and broken down by span name. transform gets its own line on the chart. So does validate . You can finally alert on one without waiting to notice it in a trace. If you'd rather not build those charts by hand, there's a dashboard export for exactly this: dashboard_span metrics.json . Import it directly into Splunk Observability Cloud and the charts are there, wired to the metric names above. The same four internal spans, now each with their own request rate, latency, and error rate — generated by importing `dashboard_span metrics.json`. Build it yourself, step by step   This works on any Collector config that already has an otlp receiver feeding a traces pipeline. We're going to change the shape of that pipeline, not what it receives or where it ultimately exports. Step 1: hook into the pipelines with the routing connector. Start from a plain pipeline like this: service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp/splunk] Rename it traces/in , and point its exporters at routing instead of the original exporter: service: pipelines: traces/in: receivers: [otlp] exporters: [routing] Now add the routing connector itself. The action: copy is the detail that matters most here — it's what forks the stream instead of diverting it: connectors: routing: default_pipelines: [traces/default] table: - condition: ContainsValue(["your-service-name"], resource.attributes["service.name"]) action: copy # keeps the span in its normal pipeline too pipelines: [traces/route-to-spanmetrics] Finally, add the two pipelines that routing feeds into. traces/default picks up where traces/in left off, with the exporters you had originally: service: pipelines: traces/default: receivers: [routing] processors: [batch] exporters: [otlphttp/splunk] traces/route-to-spanmetrics: receivers: [routing] exporters: [span_metrics]   Step 2: add the span_metrics connector   Define it alongside routing , under connectors : connectors: span_metrics: aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA metrics_flush_interval: 10s aggregation_cardinality_limit: 30 metrics_flush_interval controls how often metrics get emitted — 10 seconds is a reasonable resolution to start with. aggregation_cardinality_limit is worth setting explicitly too, since cardinality scales with the number of distinct span names you're feeding in. Then wire span_metrics into your existing metrics pipeline as an additional receiver, alongside whatever's already there: service: pipelines: metrics: receivers: [span_metrics] # add this to your existing receivers exporters: [signalfx] Restart the OpenTelemetry Collector, and check for traces.span.metrics.calls and traces.span.metrics.duration in Splunk Observability Cloud. If you want to confirm locally first, add a debug exporter to the metrics pipeline and watch the collector logs. The one place this trips people up: span_metrics has to appear on both sides — as an exporter in traces/route-to-spanmetrics , and as a receiver in metrics . Miss either side and the fork goes nowhere, silently. No error, just no metrics. Why this matters   We started with a trace that looked fine and a dashboard that didn't exist. Now fetch , validate , transform , and persist each have their own rate, error rate, and latency metrics in Splunk Observability Cloud — and getting there took a config change, not a redeploy. Nothing about the application changed. No span got relabeled into something it isn't. The trace pipeline you had is still exactly the trace pipeline you have, with one connector quietly generating metrics alongside it. That's the point of span_metrics existing in the first place: OpenTelemetry already gives you the idiomatic tool for this. You don't need to compromise your instrumentation to get visibility into what's happening inside a service. You just need to route the data you're already collecting to the place that can turn it into a metric. ... View more

Isolating Telemetry Boundaries: How to Trace Multiple WARs as Separate Services in One WildFly JVM Risk-Based Alerting (RBA) is a game-changer for reducing alert fatigue, but fine-tuning those scores manually can be a massive time sink for SOC teams. Splunk Enterprise Security 8.3 addresses this directly with Entity Risk Scoring (ERS), a feature designed to turn noisy observations into immediate, actionable intelligence. Key Takeaways The Core Problem: JVM-level agents view application servers as a single service boundary, blending spans from separate applications together. The Architecture Shift: Removing the -javaagent flag allows WildFly's native OpenTelemetry and MicroProfile subsystems to spawn isolated SDK instances per WAR. The Configuration: Define custom service.name values inside each deployment's microprofile-config.properties and scrape JVM metrics directly via the Prometheus management endpoint.     Deploy app1.war and app2.war into the same WildFly JVM, attach the default OpenTelemetry Java agent, and both applications show up as one service. The Java agent starts once at JVM level. So it gets one OpenTelemetry configuration, one resource, and one service.name . That works well for many modern Java services. But it is the wrong boundary for an application server that hosts multiple applications. Running multiple applications in one Java application server is still common. It is efficient, familiar, proven technology and often exactly how the platform is designed. But if app1.war and app2.war both report the same service.name , your observability backend cannot tell which application produced which span. That is not what we want. The good news: monitoring multiple applications in one application server as separate services is possible! The default OpenTelemetry Java agent path does not support it, but with a few targeted WildFly and application changes it will work. We will set up each deployment with its own service.name , keep Server-Timing for RUM to APM correlation, and keep JVM metrics flowing through the OpenTelemetry Collector. If you want to run it yourself, the GitHub repository contains the scripts, configuration, and a more technical guide. Architectural Problem: The JVM Boundary Mismatch The OpenTelemetry Java agent is attached to the JVM with -javaagent . For many modern Java services, that is a great default. It instruments the process early, discovers libraries, exports traces, and can collect JVM-level telemetry. When Splunk Observability is the backend, the Splunk Java agent is the logical starting point. It is based on the OpenTelemetry Java agent and adds useful Splunk-specific convenience, like automatically adding Server-Timing headers for RUM to APM correlation. But it also means the agent sees the world from the JVM boundary. And that is where the mismatch starts. In a WildFly setup with multiple WAR files, that boundary is too wide. The JVM contains multiple applications, but the agent owns one OpenTelemetry SDK and one set of resource attributes. So this kind of setup: WildFly JVM +-- app1.war `-- app2.war turns into this kind of observability model: service.name = wildfly +-- spans from app1 `-- spans from app2 That makes troubleshooting harder than it needs to be. In Splunk Observability Cloud, service identity drives filters, service maps, dashboards, alerts, and ownership. If every deployment has the same service.name , you cannot quickly tell which application produced which span. And this is where most guides stop. They assume one Java process is one service. That assumption does not hold here. Core Limitations: Why a Java Agent Cannot Solve This OpenTelemetry-based Java agents operate at JVM level. The agent starts once and applies one configuration to every deployed application. That is the core limitation. Not a bug. Just the wrong boundary for this setup. There is no setting that says: app1.war -> service.name=app1 app2.war -> service.name=app2 The agent can set otel.service.name , but that value belongs to the process. Not to each deployment inside the process. So we remove the agent from this setup. That sounds bigger than it is. We do not remove observability. We replace the agent responsibilities with pieces that understand the WildFly deployment model. Technical Solution: WildFly's MicroProfile Telemetry + OpenTelemetry Subsystems WildFly ships two cooperating subsystems that together solve this exact problem: opentelemetry — provides the OTel API/SDK runtime classes and per-deployment SDK lifecycle. microprofile-telemetry — bridges each WAR's MicroProfile Config ( otel.* properties) into its own SDK instance. Together, they create an isolated OpenTelemetry SDK instance per deployment, and each WAR reads its own configuration through MicroProfile Config. That gives us the model we need: WildFly JVM +-- app1.war -> service.name=app1 `-- app2.war -> service.name=app2 The reference setup in this post uses WildFly. Yet the same approach also applies to JBoss EAP 8. Profile Selection: Pick the Right WildFly Profile The two subsystems we need are pre-enabled in the shipped standalone-microprofile.xml profile. So instead of editing config files, we start WildFly with the -c flag: ./bin/standalone.sh -c standalone-microprofile.xml The relevant pre-enabled pieces look like this inside that profile: <extension module="org.wildfly.extension.opentelemetry"/> <extension module="org.wildfly.extension.microprofile.telemetry"/> ... <subsystem xmlns="urn:wildfly:opentelemetry:1.1"/> <subsystem xmlns="urn:wildfly:microprofile-telemetry:1.0"/> The <opentelemetry> element is intentionally empty. With nothing configured at the server level — no exporter, no sampler, no service name — the subsystem provides only the runtime classes. All SDK configuration is read per-deployment from each WAR's MicroProfile Config in the next step. That is what gives every WAR its own service.name . The default standalone.xml profile does not include the microprofile-telemetry subsystem, which is the one that bridges MicroProfile Config into the OTel SDK. Without it, per-deployment otel.service.name values are ignored and traces fall back to using the WAR filename (e.g. app1.war ) as the service name. That is why we pick the MicroProfile profile. Agent Removal: Disabling the JVM-Level Java Agent Next we remove the Java agent from standalone.conf . That means removing settings like these: JAVA_OPTS="$JAVA_OPTS -javaagent:/path/to/splunk-otel-javaagent.jar" JAVA_OPTS="$JAVA_OPTS -Dotel.service.name=wildfly" No replacement JVM options are needed for traces. A Java agent and WildFly's per-deployment OTel SDK setup cannot both own OpenTelemetry in the same JVM. The agent claims GlobalOpenTelemetry at JVM startup. Even with otel.sdk.disabled=true , it can still register a disabled SDK globally. WildFly then sees that global SDK and does not initialize its own per-deployment SDK instances. So the agent has to go. Otherwise WildFly never gets the chance to create the per-deployment SDK instances we need. Target Configurations: Setting Up Each WAR Deployment Now let's give each WAR its own service name. Create this file inside each deployment: src/main/resources/META-INF/microprofile-config.properties For app1 : otel.sdk.disabled=false otel.service.name=app1 otel.exporter.otlp.endpoint=http://localhost:4317 For app2 : otel.sdk.disabled=false otel.service.name=app2 otel.exporter.otlp.endpoint=http://localhost:4317 These three small properties create a big effect: otel.sdk.disabled=false opts this deployment in to its own OTel SDK instance. otel.service.name is the per-deployment identity we need. otel.exporter.otlp.endpoint points at the OpenTelemetry Collector. MicroProfile Config is the standard configuration mechanism used by the deployment. WildFly reads these otel.* properties per WAR, so each application gets its own resource attributes and its own exporter wiring. You can add more OpenTelemetry resource attributes in the same file. At minimum, add a version or build identifier and a deployment environment: otel.resource.attributes=service.version=1.2.0,deployment.environment=production That gives every span more useful context. service.name tells us which application emitted the span. service.version and deployment.environment tell us which version and environment produced it. Discovery Activation: Enabling CDI Beans within the WAR There is one more application-side step. WildFly's OpenTelemetry integration registers its JAX-RS tracing filter as a CDI bean. CDI, Contexts and Dependency Injection, manages bean discovery and lifecycle inside the WAR. So each deployment needs WEB-INF/beans.xml : <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="https://jakarta.ee/xml/ns/jakartaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee https://jakarta.ee/xml/ns/jakartaee/beans_4_0.xsd" version="4.0" bean-discovery-mode="all"> </beans> Without this file, the tracing filter is not discovered. And then the deployment stays silent. Frontend Correlation: Adding Server-Timing Headers Again One thing I do like about the Splunk Java agent is that it automatically adds Server-Timing response headers. Those headers expose the trace context to the browser and frontend tools, enabling RUM to APM correlation. When we remove the agent, we lose that automatic behavior. So we add it back. The fix is a small JAX-RS filter in each WAR. First add the OpenTelemetry API dependency. WildFly provides it at runtime, so the Maven scope is provided : <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-api</artifactId> <version>1.40.0</version> <scope>provided</scope> </dependency> Then add the filter: package com.example.app1; import io.opentelemetry.api.trace.Span; import io.opentelemetry.api.trace.SpanContext; import jakarta.annotation.Priority; import jakarta.ws.rs.Priorities; import jakarta.ws.rs.container.ContainerRequestContext; import jakarta.ws.rs.container.ContainerRequestFilter; import jakarta.ws.rs.container.ContainerResponseContext; import jakarta.ws.rs.container.ContainerResponseFilter; import jakarta.ws.rs.ext.Provider; @Provider @Priority(Priorities.USER) public class ServerTimingFilter implements ContainerRequestFilter, ContainerResponseFilter { private static final String SPAN_CONTEXT_PROP = "otel.span.context"; @Override public void filter(ContainerRequestContext requestContext) { SpanContext spanContext = Span.current().getSpanContext(); if (spanContext.isValid()) { requestContext.setProperty(SPAN_CONTEXT_PROP, spanContext); } } @Override public void filter(ContainerRequestContext requestContext, ContainerResponseContext responseContext) { SpanContext spanContext = (SpanContext) requestContext.getProperty(SPAN_CONTEXT_PROP); if (spanContext != null) { String traceparent = "00-" + spanContext.getTraceId() + "-" + spanContext.getSpanId() + "-01"; responseContext.getHeaders().add("Server-Timing", "traceparent;desc=\"" + traceparent + "\""); responseContext.getHeaders().add("Access-Control-Expose-Headers", "Server-Timing"); } } } This filter implements both ContainerRequestFilter and ContainerResponseFilter . During the request phase, the active span is still available. We capture its SpanContext and store it on the request. During the response phase, we read that stored context and write the Server-Timing header. The response now includes: Server-Timing: traceparent;desc="00-<traceId>-<spanId>-01" Access-Control-Expose-Headers: Server-Timing That restores the frontend correlation behavior we lost by removing the agent. Deep Visibility: Manual Instrumentation and Custom Child Spans Automatic JAX-RS tracing is useful, but application-level spans make traces much more readable. WildFly's OpenTelemetry subsystem also supports annotation-based manual instrumentation with @WithSpan and @SpanAttribute . For those annotations, add the instrumentation annotation dependency to the WAR that uses them: <dependency> <groupId>io.opentelemetry.instrumentation</groupId> <artifactId>opentelemetry-instrumentation-annotations</artifactId> <version>2.9.0</version> <scope>provided</scope> </dependency> In the sample application, app2 has a small calculation service: @ApplicationScoped public class CalculationService { @WithSpan("simulate-calculation") public long calculate(@SpanAttribute("calculation.sleep_ms") long sleepMs) { try { Thread.sleep(sleepMs); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } return sleepMs; } } The annotation creates a child span called simulate-calculation . The @SpanAttribute annotation records the sleep duration as calculation.sleep_ms . That method is called from the app2 JAX-RS resource: @GET @Produces(MediaType.TEXT_PLAIN) public Response getValue() { long sleepMs = CalculationService.randomSleepMs(); calculationService.calculate(sleepMs); return Response.ok("TWO").build(); } The result is exactly what we want. Automatic request spans, plus meaningful application spans inside the same trace. Infrastructure Setup: Configuring the OpenTelemetry Collector WildFly exports traces over OTLP gRPC, so the collector needs a gRPC receiver on port 4317 : receivers: otlp: protocols: grpc: endpoint: "0.0.0.0:4317" http: endpoint: "0.0.0.0:4318" The HTTP receiver on 4318 is still useful for other tools. The sample repository uses it for a small otel-cli test span. Telemetry Parity: Scraping JVM Metrics via Prometheus Endpoints Removing the Java agent also means removing agent-based JVM metrics collection. Fortunately, WildFly already exposes JVM and server metrics in Prometheus format at the management endpoint: http://localhost:9990/metrics So the OpenTelemetry Collector can scrape that endpoint directly: receivers: prometheus/wildfly: config: scrape_configs: - job_name: wildfly scrape_interval: 10s static_configs: - targets: ["localhost:9990"] metrics_path: /metrics service: pipelines: metrics: receivers: - prometheus/wildfly This gives us JVM metrics like heap, GC, threads, and class loading. It also gives us WildFly server metrics exposed through the management endpoint. Extra subsystem metrics depend on the statistics settings you enable in WildFly. No extra Java agent is needed. End Result: What Distributed Telemetry Looks Like Once everything is wired up, the result becomes visible in the observability backend. This is the part that matters most. Not the XML. Not the properties file. The services finally show up as separate applications. app1 and app2 shown as separate services   The same trace can still cross application boundaries. In this example, app1 calls app2 , and the trace keeps its parent-child relationship. single trace crossing app1 and app2 And if you rely on browser-to-backend correlation, the response still exposes the trace context through the Server-Timing header. server timing header with traceparent That is the result we were after. Two WARs. One WildFly JVM. Two OpenTelemetry services. The application server stays as-is, but the telemetry finally reflects the applications inside it. Deployment Variance: A Note on JBoss EAP Compatibility This setup uses WildFly 39.0.1.Final , but the same technique applies to JBoss EAP 8.0+. EAP ships the same standalone-microprofile.xml profile with the microprofile-telemetry and opentelemetry subsystems pre-enabled, so the recipe is identical: start the server with -c standalone-microprofile.xml and put the otel.* properties in each WAR's microprofile-config.properties . The Splunk agent GlobalOpenTelemetry conflict applies equally to EAP, for the same reason: a JVM-level agent that claims GlobalOpenTelemetry at startup prevents the server from creating per-deployment SDK instances. Feature Trade-offs: What About Profiling? The one Splunk-specific thing this setup does not replace is AlwaysOn Profiling. If profiling is a hard requirement, the compatible agent-based option is to run each application in a separate WildFly or EAP instance. Then each JVM can own its own full agent setup. For lightweight profiling without touching OpenTelemetry, JDK Flight Recorder is a separate option. It is built into the JDK and does not interact with the OTel SDK. Implementation Roadmap: Wrapping Up The default OpenTelemetry Java agent approach does not support separate service.name values for multiple applications inside one application server. The agent sees one JVM, so it creates one service identity. But that does not mean the architecture is unobservable. With WildFly's MicroProfile Telemetry + OpenTelemetry subsystems, each WAR gets its own SDK instance. With MicroProfile Config, each deployment gets its own service.name . With a small JAX-RS filter, we keep Server-Timing headers. And with the collector scraping WildFly's /metrics endpoint, JVM metrics keep flowing without the agent. So the final setup keeps the application server model intact: WildFly JVM +-- app1.war -> service.name=app1 `-- app2.war -> service.name=app2 That is the core takeaway. Even though the default Java agent path does not support this monitoring model, the MicroProfile profile, per-WAR config, CDI activation, a small response filter, and a collector metrics scrape give you the same operational outcome with the right service boundaries. The repository is there if you want the hands-on version with all files in place.   Don’t miss the next post. Here’s how to subscribe to this blog and get notified when new content goes live.  ... View more

This blog post is part of an ongoing series on OpenTelemetry. What happens if the OpenTelemetry collector cannot send data? Will it drop, queue in memory or on disk? Let's find out which settings are available and how they work! To check how queuing works, I've set up a small test environment. This environment consists of several data sources (postgresql, collectd and prometheus) a opentelemetry collector and a Splunk enterprise as destination. All collected data is sent to Splunk via one HEC endpoint. All internal metrics of the otel collector are sent to Splunk via another HEC endpoint. This test environment, including all example configurations can be found at my Splunk GitHub.  I will test the queueing of the otel collector by temporarily disabling the HEC endpoint which receives the collected metrics. Via the other HEC endpoint I will still receive the internal metrics, so we can see what's going on with the collector while it cannot send metrics. Queueing in the OpenTelemetry Collector Queueing is implemented in most exporters via the exporterhelper. These settings can at least be used in all exporters relevant for Splunk, sapm, splunkhec. For other exporters please check the documentation or the implementation of these exporters. Settings for queueing are thus set per exporter in the collector.yaml file. We have two types of queues available which I will be using together: In Memory Queue By default, without any configuration, data is queued in memory only. When data cannot be sent it is retried a few times (up to 5 mins. by default) and then dropped. If for any reason, the collector is restarted in this period, the queued data will be gone. If you can accept this type of data loss, you can keep the defaults or tune the queue size in collector.yaml for each exporter like this: sending_queue     queue_size (default = 5000): Maximum number of batches kept in memory before dropping. User should calculate this as num_seconds * requests_per_second / requests_per_batch  where:     num_seconds is the number of seconds to buffer in case of a backend outage     requests_per_second is the average number of requests per seconds     requests_per_batch is the average number of requests per batch (if the batch processor is used, the metric batch_send_size can be used for estimation) On Disk Persistant Queue This queue is stored to the disk, so it will persist, even when the collector is restarted. On restart the queue will be picked up and exporting is restarted.  Testing Queueing  out In my example test environment, I've set up the metrics HEC exporter to use a persistent queue which is stored on disk using the filestorage extension. From my collector.yaml: extensions: file_storage/psq: directory: /persistant_sending_queue timeout: 10s # in what time a file lock should be obtained compaction: directory: /persistant_sending_queue on_start: true on_rebound: true rebound_needed_threshold_mib: 5 rebound_trigger_threshold_mib: 3 I put the persistent queue on disk in the directory called /persistant_sending_queue. For the test, I've set some very small limits on the size of the queue on disk, 3mb and 5mb as triggers for compaction. I've also configured my splunk_hec/metrics exporter to use this queue: exporters: splunk_hec/metrics: token: ${SPLUNK_HEC_TOKEN_METRICS} endpoint: ${SPLUNK_HEC_URL} source: otel sourcetype: otel index: "metrics" max_connections: 20 disable_compression: false timeout: 10s tls: insecure_skip_verify: true retry_on_failure: enabled: true initial_interval: 1s max_interval: 300s max_elapsed_time: 120s sending_queue: storage: file_storage/psq What Happens When Data Cannot be Output? When I disable the metrics HEC input we see the following happens: data is queued data is persisted to the disk no metrics are being received What Happens When the Connection is Restored? When the metrics HEC endpoint is enabled again (see green annotation) we see the following happens: queue is drained persistent queue size on disk is slightly reduced metric data from the past is filled in, no gaps remain Even though the queue is completely drained, the on-disk persistent queue has not reduced in size much. As we didn't hit the configured high water mark of 5mb for the on-disk queue, compaction is not started. We did go below the low water mark of 3mb. This means the next time we go above 5 mb and flush the queue compaction will happen. Test Compaction I ran the same experiment again. Let the queue fill up and enable the HEC endpoint again to flush the queue. This time I went above the high water mark of 5mb and we see the on-disk file size is reduced as it should be. Considerations When Scaling up Be very mindful of your queue sizes. sending_queue.queue_size controls how much memory or disk space will be used before the collector will drop data. This value is set for each exporter, so to determine the sizing of the exporter all values should be added together. — Erwin Vrolijk, Senior Sales Engineer at Splunk ... View more

This blog post is part of an ongoing series on OpenTelemetry. OpenTelemetry is often associated with modern microservices and cloud-native applications. What happens if we apply OpenTelemetry and modern observability techniques to something completely different? WordPress is the world's most popular weblog software. And it's also an almost 20 years old monolith. What happens if we use OpenTelemetry auto tracing and the Splunk Observability cloud? Imagine you are responsible for running WordPress sites. What insights can we bring with modern tools to a popular monolith? Just by instrumenting the environment, without any changes to the WordPress code. This is exactly what I will show you, please follow along! Setup We run WordPress and the accompanying database in docker. We will not modify the WordPress code in any way. We add auto tracing to the PHP runtime as a module. We send the metrics and traces to Splunk APM. If you want to dive in directly, all code shown in this blog can be found in my WordPress Auto Instrument repo.  Add auto tracing to stock WordPress container image We use the open-source PHP tracing module https://github.com/signalfx/signalfx-php-tracing to do the tracing. In order to install this we: download the latest tracing .tgz https://github.com/signalfx/signalfx-php-tracing/releases/latest curl -s https://api.github.com/repos/signalfx/signalfx-php-tracing/releases/latest \ | grep browser_download_url \ | grep signalfx-tracing.tar.gz \ | cut -d '"' -f 4 \ | xargs curl > signalfx-tracing.tar.gz Create a Dockerfile to build our own WordPress with tracing added: FROM wordpress:php7.4 COPY $PWD/signalfx-tracing.tar.gz /tmp RUN tar -xf /tmp/signalfx-tracing.tar.gz -C / && /opt/signalfx-php-tracing/bin/post-install.sh Configure local environment We add (sensitive) environment variables to a file called .env It's best to add this file to your .gitignore and .dockerignore. The SPLUNK_ACCESS_TOKEN and SPLUNK_REALM are specific to your Splunk Observability Cloud account. Get a free trial to try this out for yourself. MYSQL_ROOT_PASSWORD=your_root_password MYSQL_USER=your_wordpress_database_user MYSQL_PASSWORD=your_wordpress_database_password SIGNALFX_SERVICE_NAME=wordpress SIGNALFX_ENDPOINT_URL=http://splunk-otel-collector:9411/api/v2/traces SPLUNK_MEMORY_TOTAL_MIB=1024 SPLUNK_ACCESS_TOKEN=<place your token here> SPLUNK_REALM=us0 Create docker compose file We base our docker compose file on the sample provided by docker: https://github.com/docker/awesome-compose/blob/master/wordpress-mysql/compose.yaml For old times' sake, we use a MySQL image, instead of the default mariadb. And we don't need to expose the database ports, we connect both containers via a dedicated network. version: '3' services: db: image: mysql:8.0 container_name: db restart: always env_file: .env environment: - MYSQL_DATABASE=wordpress volumes: - dbdata:/var/lib/mysql command: '--default-authentication-plugin=mysql_native_password' networks: - app-network For WordPress we add a build configuration, so our own Docker file is used to build the container. This adds our auto tracing to the stock WordPress container. wordpress: depends_on: - db image: traced-wordpress:latest build: ./traced-wordpress container_name: wordpress restart: always env_file: .env environment: - WORDPRESS_DB_HOST=db:3306 - WORDPRESS_DB_USER=$MYSQL_USER - WORDPRESS_DB_PASSWORD=$MYSQL_PASSWORD - WORDPRESS_DB_NAME=wordpress volumes: - wordpress:/var/www/html networks: - app-network ports: - "8080:80"  To collect metrics and traces we also add the Splunk OpenTelemetry collector container. We provide the configuration for the collector via a bind mounted file: collector.yaml. We will discuss the contents of this file later on. splunk-otel-collector: image: quay.io/signalfx/splunk-otel-collector:latest env_file: .env environment: - SPLUNK_ACCESS_TOKEN=${SPLUNK_ACCESS_TOKEN} - SPLUNK_REALM=${SPLUNK_REALM} - SPLUNK_CONFIG=/etc/collector.yaml volumes: - ./collector.yaml:/etc/collector.yaml ports: - "13133:13133" - "14250:14250" - "14268:14268" - "4317:4317" - "6060:6060" - "8888:8888" - "9080:9080" - "9411:9411" - "9943:9943" - "23456" networks: - app-network To make our WordPress config and extensions and database persistent we need to add volumes. And to enable communication between the containers we add a network: volumes: wordpress: dbdata: networks: app-network: Splunk OpenTelemetry Collector Configuration We start with a minimal configuration build using our online configurator. extensions: smartagent: bundleDir: ${SPLUNK_BUNDLE_DIR} collectd: configDir: ${SPLUNK_COLLECTD_DIR} memory_ballast: size_in_percentage: 33 receivers: zipkin: endpoint: 0.0.0.0:9411 prometheus/internal: config: scrape_configs: - job_name: otel-collector scrape_interval: 10s static_configs: - targets: - 0.0.0.0:8888 metric_relabel_configs: - source_labels: - __name__ regex: .*grpc_io.* action: drop hostmetrics: collection_interval: 10s scrapers: cpu: disk: filesystem: memory: network: load: paging: processes: processors: batch: memory_limiter: check_interval: 2s limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB} resourcedetection: detectors: - ec2 - gce - azure - ecs - system override: false resourcedetection/internal: detectors: - ec2 - gce - azure - ecs - system override: true resource/add_environment: attributes: - action: insert value: production key: deployment.environment exporters: sapm: access_token: ${SPLUNK_ACCESS_TOKEN} endpoint: ${SPLUNK_TRACE_URL} signalfx: access_token: ${SPLUNK_ACCESS_TOKEN} api_url: ${SPLUNK_API_URL} ingest_url: ${SPLUNK_INGEST_URL} sync_host_metadata: true correlation: service: extensions: - memory_ballast pipelines: metrics: receivers: - hostmetrics processors: - memory_limiter - batch - resourcedetection exporters: - signalfx metrics/internal: receivers: - prometheus/internal processors: - memory_limiter - batch - resourcedetection/internal exporters: - signalfx traces: receivers: - zipkin processors: - memory_limiter - batch - resourcedetection - resource/add_environment exporters: - sapm - signalfx To be able to identify this installation, we update the deployment.environment to uniquely identify this WordPress installation. In our case we remove production and set it to wordpress, by updating the value in this processor: resource/add_environment: attributes: - action: insert value: wordpress key: deployment.environment First start To prepare our environment we run make. This will download the PHP auto tracing library, and build our custom WordPress container. We can start our environment: % docker-compose up  When we open our browser to http://localhost:8080/ greeted with the WordPress installer. We follow the installer, and add some default configurations. When done we log in using the credentials created in the previous step. We don't like the default URL scheme so we change it to postname. Navigate to: Administration -> Settings -> Permalinks If we browse around we are already sending traces. Check in APM to see what we've already got: And we are also getting metrics from the host running our containers. Refine our collector config Traces Break down by URL We already get quite some insight into our monolith. But wouldn't it be nice if we can break down performance by URL? The visited URLs are added to the traces as the attribute http.url We can configure Splunk APM to add a break-down for this attribute, https://docs.splunk.com/observability/apm/span-tags/cmms.html. In this case, I don't want to change anything in our APM configuration, so I will solve it in the collector configuration. The OpenTelemetry collector has very powerful features to manipulate data. In this case, we need to change the span name to the contents of the attribute http.url. We add the following processor: span/endpoint: name: from_attributes: [http.url] And add the processor to the traces service: traces: receivers: - zipkin processors: - memory_limiter - batch - resourcedetection - resource/add_environment - span/endpoint exporters: - sapm - signalfx Now we can inspect our performance by URL Break down by database operation or table. In the trace view you might already have seen we collect all sql queries. Wouldn't it be great if we can also break down our performance by database table, or operation? To do this we again use the processing capabilities of the open telemetry processors. This time it is a bit more complicated, as we need to extract the operation and database table from the query, and put it into a new attribute. We extract the db operation using this regex: (?ms)(?P<dboperation>\w+). This is a very naive regex to simply get the first word of the query. Which works surprisingly well. Extracting the tables is a bit trickier when JOIN's are involved. For the sake of simplicity I only extract the base table, and exclude any table joined to it for now. The regex to grab the word after FROM is (?ms).*FROM\s`?(?P<dbtable>[^\s`]+). These regexes use named capturing groups to extract data. dboperation and dbtable. The attributes we need are named db.operation and db.sql.table. Unfortunately regex does not support a dot in the name of a capturing group. So we need to rename the extracted attribute. Renaming attributes in the OpenTelemetry collector is done in two steps. Duplicate the attribute to the new name. And delete the original attribute. This brings us to the following configuration. attributes/db: include: match_type: strict attributes: - key: component value: mysqli actions: - key: resource.name pattern: (?ms)(?P\w+) action: extract - key: db.operation from_attribute: dboperation action: insert - key: dboperation action: delete - key: resource.name pattern: (?ms).*FROM\s`?(?P[^\s`]+) action: extract - key: db.sql.table from_attribute: dbtable action: insert - key: dbtable action: delete You see the extract, insert, delete action twice. For both the table and operation. At the top you see we restrict these transformations to traces that have the attribute component set to mysqli. This way we are sure we only manipulate database-related traces. We also need to add this processor to the traces service: traces: receivers: - zipkin processors: - memory_limiter - batch - resourcedetection - resource/add_environment - attributes/db - span/endpoint exporters: - sapm - signalfx Restarting the OpenTelemetry collector will show we now have nice database breakdowns available: Mysql metrics Now that we have quite some insight into what WordPress and the database are doing, wouldn't it be nice to see some more details of our database? We can get that by easily adding a receiver for MySQL to our collector config: smartagent/mysql: type: collectd/mysql host: db port: 3306 databases: - name: wordpress username: ${MYSQL_USER} password: ${MYSQL_PASSWORD} extraDimensions: service.name: mysql-service And add this receiver to our metrics pipeline service: pipelines: metrics: receivers: - hostmetrics - smartagent/mysql This makes the default MySQL dashboard light up: Final result Now we have fully instrumented our WordPress monolith with modern tooling, without any changes to the WordPress application itself! Amongst other things, we can see latency per URL, inspect slow queries, see which URLs map to which queries, and inspect our most important database and host metrics. Pretty impressive if you ask me! Further improvements All techniques mentioned in this article are done without any changes to the WordPress or template code. If needed we could of course go further. We could add our Real User Monitoring to the front end, so we get deep insight into how the pages render on the client devices. This could point us to performance problems and bugs in the javascript and CSS used by the WordPress templates or posts. To further instrument the backend we can add manual tracing statements to our WordPress plugins. This will show us in great detail which plugin contributes how much to the total back end loading time. — Erwin Vrolijk, Sr. Solution Engineer ... View more