Community Blog
Get the latest updates on the Splunk Community, including member experiences, product education, events, and more!

Capturing HTTP Headers As Span Attributes

jplumb
Splunk Employee
Splunk Employee

This blog post is part of an ongoing series on OpenTelemetry.

The github repository for this article is here.

We often get asked how to copy http request/response headers into OpenTelemetry span attributes. The java instrumentation agent does not do this by default, primarily because it risks leaking sensitive data. Fortunately, there is a way to instruct the java agent to collect http headers and attach them to your spans.

This project is a quick demonstrative example of capturing http headers as span attributes.

The main application is HttpHeaderAttributesMain. This application spins up a small HTTP server on port 8182, and a client that sends a request to the server once a second.

This is an unusual ab/use of HTTP and is only intended to demonstrate how to tell the java instrumentation to copy headers into attributes.

Client request

The client sends a POST request with a demeanor HTTP header and the name of a person in the request body.

POST /greeting HTTP/1.1
Content-Length: 8
Host: localhost:8182
User-Agent: Java-http-client/11.0.9.1
Content-Type: text/plain
demeanor: TIRED

Benjamin

Server responses:

The server responds with an originator header whose value is the uppercase version of the person's name. The body contains the person's name and an emoji corresponding to their demeanor.

HTTP/1.1 200 OK
Date: Thu, 15 Dec 2022 21:39:27 GMT
Content-Type: text/plain
originator: BENJAMIN
Transfer-Encoding: chunked
Server: Jetty(9.4.48.v20220622)

Benjamin => 🥱

Running

To build and run the example, simply run the gradle build target run. You must have java 11+ installed and an otel collector running on localhost.

./gradlew run

Configuration

Configuration is provided to the java agent via system properties passed on the commandline. No manual code changes required!

The upstream community documentation for this feature is here.

If you look in the build.gradle.kts file, you will notice several JVM commandline args. These are used to wire up the java agent, and provide our custom configuration:

-javaagent:splunk-otel-javaagent-1.18.0.jar
-Dotel.javaagent.debug=true
-Dotel.resource.attributes=deployment.environment=http-testenv
-Dotel.service.name=HttpHeaderAttributes
-Dotel.instrumentation.http.capture-headers.client.request=demeanor
-Dotel.instrumentation.http.capture-headers.client.response=originator
-Dotel.instrumentation.http.capture-headers.server.request=demeanor,user-agent
-Dotel.instrumentation.http.capture-headers.server.response=originator


The last 4 are the interesting ones. These tell the agent which headers to capture. Specifically, we are asking the java agent to capture these 4 headers:

  • client instrumentation - outgoing request - demeanor
  • client instrumentation - incoming response - originator
  • server instrumentation - incoming request demeanor and user-agent
  • server instrumentation - outgoing response - originator

If your application is purely a client or purely a server, you would probably only need half of this, but we've provided both directions on both sides for demonstration.

Results

In the APM trace view, we have generated a simple trace with one client span and one server span:

jplumb_3-1671145513275.png

If we expand the topmost client span, we can view the relevant attributes:

jplumb_4-1671145513276.png

We see that the span contains two new attributes that would not have existed otherwise: http.request.header.demeanor and http.response.header.originator.

If we now expand the server span, we can see the new attributes there as well:

jplumb_5-1671145513278.png

The three additional attributes are http.request.header.demeanor, http.request.header.user_agent, and http.response.header.originator.

Mappings

HTTP headers that are turned into attributes are namespaced/prefixed with one of:

  • http.request.header.
  • http.response.header.

In addition to the prefix being added header names will have hyphens converted to underscores (see User-Agent becoming user_agent in the above example).

Furthermore, because HTTP headers may be repeated, the values are treated as a list. The resulting attribute value is surrounded with square brackets and the values are concatenated with commas.

— Jason Plumb, Principal Software Engineer, OpenTelemetry Java Contributor

Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...