Today I have upgraded from Splunk-6.3.3 to Splunk-6.4.0 and web interface has stopped working. I get HTTP 500 Internal Server Error each time I try to open any page of Splunk.
I am using SSO, but it doesn't work even after disabling it. CLI works fine, I can login an make a request there.
Log of clean try to login:
2016-04-06 13:33:31,465 ERROR [5704c9db347f5e5c2cae50] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:31,467 INFO [5704c9db347f5e5c2cae50] decorators:362 - require_login - no splunkd sessionKey variable set; cherrypy_session=f7881bf5cdb6e18c9d603e35261fb4d6eec6b9d1 request_path=/en-US/
2016-04-06 13:33:31,468 INFO [5704c9db347f5e5c2cae50] decorators:383 - require_login - redirecting to login
2016-04-06 13:33:31,745 ERROR [5704c9db827f5e5c321d90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:31,971 ERROR [5704c9db827f5e5c321d90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:32,455 ERROR [5704c9dc397f5e5c2caf90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:36,591 ERROR [5704c9e0547f5e5c2ca390] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:36,817 ERROR [5704c9e0547f5e5c2ca390] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:37,042 DEBUG [5704c9e0547f5e5c2ca390] _cplogging:55 - [06/Apr/2016:13:33:37] HTTP Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 606, in respond
cherrypy.response.body = self.handler()
File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 25, in __call__
return self.callable(*self.args, **self.kwargs)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 38, in rundecs
return fn(*a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 118, in check
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 167, in validate_ip
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 407, in handle_exceptions
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 462, in apply_cache_headers
response = fn(self, *a, **kw)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py", line 176, in login
sessionKey = splunk.auth.getSessionKey(username, password, hostPath=self.splunkd_urlhost, newPassword=newpassword)
File "/opt/splunk/lib/python2.7/site-packages/splunk/auth.py", line 31, in getSessionKey
serverResponse, serverContent = rest.simpleRequest(uri, postargs=args)
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 516, in simpleRequest
raise splunk.AuthorizationFailed(extendedMessages=uri)
AuthorizationFailed: [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/auth/login
2016-04-06 13:33:37,042 INFO [5704c9e0547f5e5c2ca390] _cplogging:55 - [06/Apr/2016:13:33:37] HTTP
Request Headers:
USER-AGENT: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
REFERER: https://my-noc-instance.domain.com/splunk/en-US/account/login?return_to=%2Fsplunk%2Fen-US%2F
Content-Type: application/x-www-form-urlencoded
X-FORWARDED-HOST: my-noc-instance.domain.com
ACCEPT-LANGUAGE: en-US,en;q=0.5
HOST: my-noc-instance.domain.com:8000
ACCEPT-ENCODING: gzip, deflate
X-FORWARDED-SERVER: my-noc-instance.domain.com
ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
X-FORWARDED-FOR: 10.10.10.10
CONNECTION: Keep-Alive
COOKIE: cval=2069472843; session_id_8000=f7881bf5cdb6e18c9d603e35261fb4d6eec6b9d1; splunkweb_uid=57DDAC14-229E-4E10-80D8-F573359D9066
DNT: 1
X-REMOTE-USER: username@DOMAIN.COM
Content-Length: 85
VIA: 1.1 my-noc-instance.domain.com
Remote-Addr: 127.0.0.1
... View more