Community Blog
Get the latest updates on the Splunk Community, including member experiences, product education, events, and more!

Capturing HTTP Headers As Span Attributes

Splunk Employee
Splunk Employee

This blog post is part of an ongoing series on OpenTelemetry.

The github repository for this article is here.

We often get asked how to copy http request/response headers into OpenTelemetry span attributes. The java instrumentation agent does not do this by default, primarily because it risks leaking sensitive data. Fortunately, there is a way to instruct the java agent to collect http headers and attach them to your spans.

This project is a quick demonstrative example of capturing http headers as span attributes.

The main application is HttpHeaderAttributesMain. This application spins up a small HTTP server on port 8182, and a client that sends a request to the server once a second.

This is an unusual ab/use of HTTP and is only intended to demonstrate how to tell the java instrumentation to copy headers into attributes.

Client request

The client sends a POST request with a demeanor HTTP header and the name of a person in the request body.

POST /greeting HTTP/1.1
Content-Length: 8
Host: localhost:8182
User-Agent: Java-http-client/
Content-Type: text/plain
demeanor: TIRED


Server responses:

The server responds with an originator header whose value is the uppercase version of the person's name. The body contains the person's name and an emoji corresponding to their demeanor.

HTTP/1.1 200 OK
Date: Thu, 15 Dec 2022 21:39:27 GMT
Content-Type: text/plain
originator: BENJAMIN
Transfer-Encoding: chunked
Server: Jetty(9.4.48.v20220622)

Benjamin => 🥱


To build and run the example, simply run the gradle build target run. You must have java 11+ installed and an otel collector running on localhost.

./gradlew run


Configuration is provided to the java agent via system properties passed on the commandline. No manual code changes required!

The upstream community documentation for this feature is here.

If you look in the build.gradle.kts file, you will notice several JVM commandline args. These are used to wire up the java agent, and provide our custom configuration:


The last 4 are the interesting ones. These tell the agent which headers to capture. Specifically, we are asking the java agent to capture these 4 headers:

  • client instrumentation - outgoing request - demeanor
  • client instrumentation - incoming response - originator
  • server instrumentation - incoming request demeanor and user-agent
  • server instrumentation - outgoing response - originator

If your application is purely a client or purely a server, you would probably only need half of this, but we've provided both directions on both sides for demonstration.


In the APM trace view, we have generated a simple trace with one client span and one server span:


If we expand the topmost client span, we can view the relevant attributes:


We see that the span contains two new attributes that would not have existed otherwise: http.request.header.demeanor and http.response.header.originator.

If we now expand the server span, we can see the new attributes there as well:


The three additional attributes are http.request.header.demeanor, http.request.header.user_agent, and http.response.header.originator.


HTTP headers that are turned into attributes are namespaced/prefixed with one of:

  • http.request.header.
  • http.response.header.

In addition to the prefix being added header names will have hyphens converted to underscores (see User-Agent becoming user_agent in the above example).

Furthermore, because HTTP headers may be repeated, the values are treated as a list. The resulting attribute value is surrounded with square brackets and the values are concatenated with commas.

— Jason Plumb, Principal Software Engineer, OpenTelemetry Java Contributor

Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...