Splunk 6 & Chrome incorrectly sets Authorization header causing user auth to fail

[clayton_bell_ag]

Our setup

• Splunk 6 (recently upgrade from 5 to 6) on ubuntu 64bit
• Apache2 mod_proxy SSL acting as a reverse proxy to the Splunk Web UI
• Apache2 Basic Auth so that all splunk related URIs require an authenticate username/password from a htpassword file

Since upgrading to Splunk 6, some URIs refuse to authenticate. The only change to the server has been to upgrade Splunk from v5.0.4-172409 to v6.0.1-189883. No other changes were made.

Since the upgrade, Chrome incorrectly sets the Autorization header value to the string "Splunk" (without the quotes) on the faulty URIs. It should be a string of the form "Basic < base64_string >". This happens specifically on each dashboard widget: the main dashboard URI authenticates fine, but the subsequent Ajax calls that populate each dashlet have the incorrect header value set to "Splunk".

Specially, when going to https://< server >/en-GB/app/search/< dashboard >
(note that I've generalized url's and ip addresses) will cause apache to log the following error

[Mon Feb 14 14:14:14 2014] [error] [client XX.XX.XX.XX] client used wrong authentication scheme: /en-GB/splunkd/__raw/servicesNS/admin/search/saved/searches/< saved_search >, referer: https://< server >/en-GB/app/search/< dashboard >

This line is logged for each dashlet on the dashboard that is trying to be viewed.
Note that you can access the saved search just fine by typing in the URL https://< server >/en-GB/splunkd/__raw/servicesNS/admin/search/saved/searches/< saved_search > or by using the Web UI to browse to the saved search and run it from there. Its possibley something javascript related on some ajax calls not maintaining the Authorization header. No amount of typing in the apache htpassword username/password will result in the broken ajax calls working as the header value is always Splunk.

The issue does not occur when using Splunk 5+Chrome/Firefox.

The issue does not occur using Splunk 6 and Firefox.

The issue does occur with Splunk 6 and Chrome.

Here's the apache config snippet we've used for both Splunk 5 and Splunk 6

<VirtualHost *:443>
  ServerName splunk.blah.com

  SSLEngine on
  SSLCertificateFile /etc/apache2/ssl/wild.blah.com.crt   # our certs are for *.blah.com
  SSLCertificateKeyFile /etc/apache2/ssl/wild.blah.com.key
  SSLCertificateChainFile /etc/apache2/ssl/our_providers_ca_bundle.pem

  SSLProxyEngine on
  ProxyPass / https://127.0.0.1:8000/ status=I retry=0
  ProxyPassReverse / https://127.0.0.1:8000/
  ProxyPreserveHost On

  <Location "/">
    AuthType Basic
    AuthName "Auth Required"
    AuthUserFile "/our/htpasswd.file"
    Require valid-user

    Order allow,deny
    Allow from all
  </Location>

</VirtualHost>

Apache Digest Auth produces the same result: incorrect Authorization header and still logs the message "client used wrong authentication scheme"

I'm using Chrome v 32.0.1700.107 on OSX 10.8.2. Googling for the issue shows a few other people have had identical issues with Splunk 6, apache auth & specifically Chrome 30.0.1599.66 on ubuntu & Chrome 31.0.1650.63 on OSX

I'm hoping some one from development can comment.

[bosburn_splunk]

This is a known bug and will be fixed in the next release..

[koudis]

We have the same issue on 6.0.3.

[erikda · Accepted solution]

Try:

<LocationMatch "__raw">
Satisfy Any
Allow from all
RequestHeader unset REMOTE_USER
</LocationMatch>

[clayton_bell_ag]

Excellent that resolves the issue. Note for other users that any url with __raw in it will bypass apache auth. Sounds bad? Not entirely..

URLs with __raw in it will be unchallenged by Apache however Splunk will return a HTTP 401 as the user will be missing the required session cookie. The only way to get the required session cookie is to visit the landing page of Splunk. However, the landing page is password protected via Apache. So external attackers can still touch the splunk web UI but should find it hard to work around this.

Thanks you for your time.