How do I fix Apache and Nginx dropped headers needed for AppDynamics?

[#1]

Apache and Nginx have a known issue that drops headers containing underscores.
How can I fix these headers, since they're needed for correlation purposes?

Problem

Recently, Apache and Nginx began dropping headers that include underscores in their names, like “ADRUM_1”.

AppDynamics needs this type of header name for correlation purposes. So, if Apache and Nginx interpret header names with underscores as broken and thus drop them, the necessary correlation won’t take place.

Verification

Mobile app identification happens with a header with an underscore (e.g., adrum_1). For browsers, it identifies by ADRUM: isajax:true (which does not have an underscore).

For BRUM correlation, the headers look like this:

< ADRUM_0: g%3A89b16a83-9478-40e9-a625-bf101d57c6f018
< ADRUM_1: i:563
< ADRUM_2: n:customer1_cef43e07-de57-4fce-939a-54a6c5dcdf56
< ADRUM: isAjax:true

Whereas the correlation header in case of Mobile will appear as follows:

< ADRUM_0: clientRequestGUID:3de4f995-e0ba-4d59-971f-d653e1ea44cf
< ADRUM_1: globalAccountName:customer1_7cc5e33b-579e-4e57-b302-b1eb63b30a0c
< ADRUM_1: isMobile:true

Broken Correlation Headers

You can check these headers with the following command on a valid endpoint of your application:

curl -v --header"ADRUM: isAjax:true" --header "ADRUM_1: isMobile:true" <http[s]://<endpoint-of-your-application-for-which-you-do-not-see-correlation>

The ADRUM_1: isMobile:true header is used to identify that the request is coming from a mobile app, whereas ADRUM: isAjax:true means it is for browser correlation (no underscore). But due to a known issue with Apache and Nginx, this header is stripped off due to the underscore in the header name. These are known as broken headers.

Underscore Header Work-around in Apache and Nginx

• Apache work-around
• Nginx work-around

Apache work-around

Apache is silently dropping off headers (including underscores). This is documented by Apache for Apache 2.4 where they describe the following workaround: 

1. Add the following in the httpd.conf:

SetEnvIfNoCase ^ADRUM.1$ ^(.*)$ fix_adrum_1=$1 RequestHeader set ADRUM-1 %{fix_adrum_1}e env=fix_adrum_1

2. Restart your apache server. You will see that these are no more dropped headers.

Nginx work-around

NGINX drops HTTP headers with underscores silently. To correct this issue, change the nginx.conf file to add the following:

underscores_in_headers on;

Once you complete these workarounds, headers with underscores won’t be dropped by the webserver.

See Nginx’ configuration pitfalls documentation. 

Resources

Overview of new features in Apache HTTP Server 2.4

Missing (disappearing) HTTP Headers