SSO works with anything but REMOTE_USER

[Wilcooley]

This is maddening but at this point, I know how to work around it but not why I am seeing it. I am setting up SSO and, as far as I can tell, REMOTE_USER is being ignored or scrubbed within splunkweb. Splunk is 4.3.2; I think I saw pretty much the same behaviour when I was first setting up SSO on another host a couple of years ago; it was 4.0 or 4.1 at the time.

I have looked at a number of the other questions related to this topic and some seem to kinda dance around this issue (notice the bits at the bottom about REMOTE-USER):

how-do-i-make-single-sign-on-work-with-mod_proxy

Does anybody know what's going on here? I have a working setup now, but I believe this is a bug that causes more than a little confusion.

I have Apache with mod_proxy setup to reverse-proxy the requests to localhost:8000. Through several methods, I have it passing 4 HTTP request headers: Cas-User, REMOTE-USER, REMOTE_USER and X-Forwarded-User. I have verified this with tcpdump:

# tcpdump -s0 -A -i lo port 8000
...
GET /en-US/debug/sso HTTP/1.1
Host: localhost:8000
Cache-Control: max-age=0
Pragma: no-cache
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.162 Safari/535.19
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: UTF-8,*;q=0.5
Cookie: MOD_AUTH_CAS=***; session_id_8000=***
CAS-User: wcooley
REMOTE_USER: wcooley
REMOTE-USER: wcooley
X-Forwarded-User: wcooley
Via: 1.1 splunkhost.example.com
X-Forwarded-For: x.x.x.x
X-Forwarded-Host: splunkhost.example.com
X-Forwarded-Server: splunkhost.example.com
Connection: Keep-Alive

trustedIP in both server.conf and web.conf are 127.0.0.1.

If remoteUser in web.conf is set to Cas-User, REMOTE-USER or X-Forwarded-User, SSO works. The SSO debug has, for example:

Remote User HTTP Header Cas-User
Value of Cas-User   wcooley

If remoteUser is set to REMOTE_USER or is unset (yes, I tested both), then SSO does not work:

Remote User HTTP Header REMOTE_USER
Value of REMOTE_USER    Not set. SSO may not be enabled or you may not be accessing Splunk via your proxy server.

And I am redirected to the login page for other requests. Also, "Other HTTP Headers" has the following:

Accept  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset  UTF-8,*;q=0.5
Accept-Encoding gzip,deflate,sdch
Accept-Language en-US,en;q=0.8
Cas-User        wcooley
Connection      Keep-Alive
Cookie  ...
Host    localhost:8000
Remote-Addr     127.0.0.1
Remote-User     wcooley
User-Agent      Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.162 Safari/535.19
Via     1.1 splunkserver.example.com
X-Forwarded-For x.x.x.x
X-Forwarded-Host        splunkserver.example.com
X-Forwarded-Server      splunkserver.example.com
X-Forwarded-User        wcooley

Notice that REMOTE_USER is not set, but REMOTE-USER is. I tried separately disabling REMOTE_USER and REMOTE-USER, but in all three cases (both enabled, only one of each enabled) the result appears to be the same.

Just to ensure that my mod_auth_cas SSO provider wasn't secretly doing something obnoxious, I switched to basic auth in Apache, with the same results (except, of course, Cas-User wasn't included in the headers).

*whew*

[vidyadharms]

I am trying SSO using IIS 8.5 as reverse proxy, ARR v3.0 and URL rewrite Module 2 however when I hit the IIS page, it redirets me to Splunk login page and SSO does't work.

I could see below lines in web_services.log file.
2016-09-13 04:34:07,611 INFO [57d7e42f9addec85acc0] decorators:383 - require_login - redirecting to login
2016-09-13 04:53:36,280 INFO [57d7e8c046ddec85a128] decorators:362 - require_login - no splunkd sessionKey variable set; cherrypy_session=3a5162816f62d2fc5a7fa1ce48d872b83ee94e20 request_path=/en-US/
2016-09-13 04:53:36,282 INFO [57d7e8c046ddec85a128] decorators:383 - require_login - redirecting to login

Also in SSO debug page of splunk, I see a blank value for X-Remote-User header variable.

Can someone help me to resolve it?

Thanks in advance.

[Wilcooley]

As of 5.0.2, the docs are... different. "Troubleshoot Splunk SSO" says, "Splunk is configured to accept the remote header value of X_REMOTE_USER, which is the default for most proxies." http://docs.splunk.com/Documentation/Splunk/5.0.2/Security/TroubleshootSplunkSSO

"Configure ..." in that same doc says, "The default Splunk header used is REMOTE_USER, ...". And web.conf.spec agrees.

[wsw70]

@Simon: I did not test extensively but if the "special character" not to use would be the underscore, it would clash with splunk's documentation (where REMOTE_USER is mentioned). Unfortunately it appears that it could be the case (I do not know if a header like HELLO_WORLD would work or is it just REMOTE_USER which is specifically a problem).
Anyway - this is a bug and I will open a ticket with splunk about that.

[anshu2812]

I got SSO working with splunk using ARR v2.5 and URL rewriter 2.0 and it works as a charm.

[vidyadharms]

Can you please share the steps followed?

[Wilcooley]

Using REMOTE_USER as the HTTP header or something else?

(To save everyone else the trouble of figuring out what "ARR v2.5 and URL rewriter 2.0" are: They're part of Microsoft IIS.)

[Simon]

My observation is, that Splunk (maybe only in certain version) only accepts HTTP-headers for SSO not containing any special character. At least, it works for me when using something like "SPLUNKUSER"

[wsw70]

Same problem with splunk 5.0.1. Thanks for the solution, I would never have guessed that REMOTE_HOST does not work (I posted my config at http://splunk-base.splunk.com/answers/75090/sso-configuration-example-of-an-apache-proxy-to-cas, cross-referencing your solution on the way)

[fervin]

I can't get REMOTE-USER to work, either. Our config worked before 4.3 .. LDAP works fine, SSO debug page says SSO will be used, but the search app insists user=UNKNOWN_USER. I really want to upgrade to 4.3 but here we are at 4.3.3 with no resolution. Come on Splunk, fix this!

[wageof]

I can confirm that REMOTE_USER doesn't work

[Wilcooley]

Perhaps I should ask the follow-up question: Does this actually work for other people?