topic Re: Issues with thread management session? in Monitoring Splunk

Issues with thread management session?

sloshburch — Mon, 11 Feb 2013 14:11:30 GMT

I saw this issue on multiple browsers.

I have splunk fronted by four reverse proxy servers running IBM HTTP Server 7.0.0.17 (Apache 2.2.8) presenting SiteMinder as a SSO agent.

To debug/verify the setup on each individual web server, I navigate to http://.company.com:. This prompts me to log in as expected. I am then able to use any non search related feature in splunk. By that, I mean that I can navigate to the manager but if I navigate to search I am redirected to the native splunk login page.

I see this activity corresponding in the logs:

2013-02-11 09:10:00,976 INFO [5118fbb8ea4e379d0] cached:77 - memoized decorator used on function <function getEntities at 0x4258cf8> with non hashable arguments 2013-02-11 09:10:01,336 INFO [5118fbb9465007c90] cached:77 - memoized decorator used on function <function getEntities at 0x4258cf8> with non hashable arguments 2013-02-11 09:10:01,774 INFO [5118fbb9465007c90] view:1070 - PERF - viewTime=0.2188s templateTime=0.2196s 2013-02-11 09:10:03,112 ERROR [5118fbbb175115150] utility:125 - [HTTP 401] Client is not authenticated Traceback (most recent call last): File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/utility.py", line 123, in parse_time parsed = times.splunktime2Iso(ts) File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/times.py", line 173, in splunktime2Iso serverStatus, serverResp = splunk.rest.simpleRequest('/search/timeparser', getargs=getargs) File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 452, in simpleRequest raise splunk.AuthenticationFailed AuthenticationFailed: [HTTP 401] Client is not authenticated 2013-02-11 09:10:03,365 WARNING [5118fbbb59563d2d0] util:1255 - CSRF form_key mismatch received=14808273786252083278 expected=None 2013-02-11 09:10:03,367 WARNING [5118fbbb59563d2d0] decorators:87 - CSRF: validation failed because client XHR did not include proper header 2013-02-11 09:10:03,376 WARNING [5118fbbb5c50e0890] util:1255 - CSRF form_key mismatch received=14808273786252083278 expected=None 2013-02-11 09:10:03,378 WARNING [5118fbbb5c50e0890] decorators:87 - CSRF: validation failed because client XHR did not include proper header 2013-02-11 09:10:03,570 WARNING [5118fbbb8f4caa190] util:1255 - CSRF form_key mismatch received=14808273786252083278 expected=None 2013-02-11 09:10:03,570 WARNING [5118fbbb8f4caa190] decorators:87 - CSRF: validation failed because client XHR did not include proper header 2013-02-11 09:10:03,738 WARNING [5118fbbbb750eb310] util:1255 - CSRF form_key mismatch received=14808273786252083278 expected=None 2013-02-11 09:10:03,740 WARNING [5118fbbbb750eb310] decorators:87 - CSRF: validation failed because client XHR did not include proper header 2013-02-11 09:10:03,742 WARNING [5118fbbbb750eb310] decorators:95 - CSRF: skipping 401 redirect response because endpoint did not request protection

This does NOT occur when navigating to splunk with my dns/VIP which load balances amongst the four web servers with a sticky session.

I am aware that there is a domain requirement on my SOS policy that requires *.company.com.

I'm not sure how to phrase my question, or if I'm using the right terms. My SOS engineer suggests I research how the session is passed from thread to thread (given my interpretation that search spawns another thread).

Any idea why I am being redirected back to the login page?

Re: Issues with thread management session?

dart — Mon, 11 Feb 2013 21:25:14 GMT

Try hitting the /debug/sso endpoint and see what you get back. It should tell you what is being passed by the Proxy.

Re: Issues with thread management session?

sloshburch — Mon, 28 Sep 2020 13:18:00 GMT

Thanks. I checked there but I'm not confident with the results I've found. 😞

I did find the following:
- In the working scenario, I have a Cookie with a session_id_39002. This is missing from the broken one. 39002 is my splunk httpport.
- In the working scenario, X-Forwarded-Host is splunk.company.com, hostname.company.com:listenport in the broken scenario.
- As expected, the User listed in Server info: footer of the working scenario is my user ID, but 'UNKNOWN_USER' in the broken scenario.
All other values match.

Re: Issues with thread management session?

dart — Tue, 12 Feb 2013 15:08:02 GMT

So you don't have anything in the Value of Remote-User section or Is the incoming request IP in splunkweb's list of trustedIPs?

Re: Issues with thread management session?

sloshburch — Wed, 13 Feb 2013 22:13:25 GMT

I found the issue - not so much an issue, more a feature.

tools.sessions.secure in web.conf, by default, restricts session to only https. By connecting directly to my reverse proxy, I was not using an https connection, but rather an http.

I proved this by setting tools.sessions.secure = False and was able to connect.

Re: Issues with thread management session?

lakshman237 — Tue, 30 Jun 2015 06:19:18 GMT

Hi - I have a similar issue. did setting the tools.session.secure = False alone was enough?

Re: Issues with thread management session?

sloshburch — Tue, 30 Jun 2015 12:08:05 GMT

It's been a long time and I honestly don't recall anymore 😞 Give it a shot and see? Let us know what you find out?