Installation
Highlighted

500 Internal Server Error after upgrading from 6.5.3 to 7.1.3.

Explorer

Today I have upgraded from Splunk-6.3.3 to Splunk-6.4.0 and web interface has stopped working. I get HTTP 500 Internal Server Error each time I try to open any page of Splunk.

I am using SSO, but it doesn't work even after disabling it. CLI works fine, I can login an make a request there.

Log of clean try to login:

2016-04-06 13:33:31,465 ERROR   [5704c9db347f5e5c2cae50] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:31,467 INFO    [5704c9db347f5e5c2cae50] decorators:362 - require_login - no splunkd sessionKey variable set; cherrypy_session=f7881bf5cdb6e18c9d603e35261fb4d6eec6b9d1 request_path=/en-US/
2016-04-06 13:33:31,468 INFO    [5704c9db347f5e5c2cae50] decorators:383 - require_login - redirecting to login
2016-04-06 13:33:31,745 ERROR   [5704c9db827f5e5c321d90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:31,971 ERROR   [5704c9db827f5e5c321d90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:32,455 ERROR   [5704c9dc397f5e5c2caf90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:36,591 ERROR   [5704c9e0547f5e5c2ca390] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:36,817 ERROR   [5704c9e0547f5e5c2ca390] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:37,042 DEBUG   [5704c9e0547f5e5c2ca390] _cplogging:55 - [06/Apr/2016:13:33:37] HTTP Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 606, in respond
    cherrypy.response.body = self.handler()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 25, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 38, in rundecs
    return fn(*a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 118, in check
    return fn(self, *a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 167, in validate_ip
    return fn(self, *a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 407, in handle_exceptions
    return fn(self, *a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 462, in apply_cache_headers
    response = fn(self, *a, **kw)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py", line 176, in login
    sessionKey = splunk.auth.getSessionKey(username, password, hostPath=self.splunkd_urlhost, newPassword=newpassword)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/auth.py", line 31, in getSessionKey
    serverResponse, serverContent = rest.simpleRequest(uri, postargs=args)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 516, in simpleRequest
    raise splunk.AuthorizationFailed(extendedMessages=uri)
AuthorizationFailed: [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/auth/login

2016-04-06 13:33:37,042 INFO    [5704c9e0547f5e5c2ca390] _cplogging:55 - [06/Apr/2016:13:33:37] HTTP
Request Headers:
  USER-AGENT: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
  REFERER: https://my-noc-instance.domain.com/splunk/en-US/account/login?return_to=%2Fsplunk%2Fen-US%2F
  Content-Type: application/x-www-form-urlencoded
  X-FORWARDED-HOST: my-noc-instance.domain.com
  ACCEPT-LANGUAGE: en-US,en;q=0.5
  HOST: my-noc-instance.domain.com:8000
  ACCEPT-ENCODING: gzip, deflate
  X-FORWARDED-SERVER: my-noc-instance.domain.com
  ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  X-FORWARDED-FOR: 10.10.10.10
  CONNECTION: Keep-Alive
  COOKIE: cval=2069472843; session_id_8000=f7881bf5cdb6e18c9d603e35261fb4d6eec6b9d1; splunkweb_uid=57DDAC14-229E-4E10-80D8-F573359D9066
  DNT: 1
  X-REMOTE-USER: username@DOMAIN.COM
  Content-Length: 85
  VIA: 1.1 my-noc-instance.domain.com
  Remote-Addr: 127.0.0.1

 

Labels (1)
Highlighted

Re: 500 Internal Server Error after upgrading to 6.4.0

Splunk Employee
Splunk Employee

There have been other reports of this. If you have an active support agreement, please open a case so Support can track the issues across customer sites.

We are developing workaround information and will update the release notes soon.

0 Karma
Highlighted

Re: 500 Internal Server Error after upgrading to 6.4.0

Splunk Employee
Splunk Employee

The release notes are updated with the following workaround information:

Option 1: Export the following setting into your environment prior to starting Splunk Enterprise: NO_PROXY="127.0.0.1,localhost,[::1]"

export NO_PROXY="127.0.0.1,localhost,[::1]"
or add it to $SPLUNK_HOME/etc/splunk-launch.conf and restart Splunk Enterprise.

Option 2: Unset the proxy on the Linux system and restart Splunk Enterprise:

`echo $http{,s}proxy
unset http{,s}
proxy'

If the proxy is set in splunk-launch.conf but is not actually used or needed, remove the setting and restart.

View solution in original post

Highlighted

Re: 500 Internal Server Error after upgrading to 6.4.0

Explorer

Works now. Thanks!

0 Karma
Highlighted

Re: 500 Internal Server Error after upgrading to 6.4.0

Splunk Employee
Splunk Employee

Corrected option 2 per the Answer and comment below.

0 Karma
Highlighted

Re: 500 Internal Server Error after upgrading to 6.4.0

Path Finder

Hi,

In which file we need to put the below code

echo $http{,s}proxy
unset http{,s}
proxy'

or where we need to execute this code?

0 Karma
Highlighted

Re: 500 Internal Server Error after upgrading to 6.4.0

Path Finder

Yes, it's a Unix command. You have to execute it in the shell. Of course you can wrap it in shell script, the "./splunk start" at the end.
But why not take 6.4.1 or 6.4.2? This has only been an issue in 6.4.0.

0 Karma
Highlighted

Re: 500 Internal Server Error after upgrading to 6.4.0

Path Finder

Thanks. But this problem also happens in 6.4.2 where the only page affected is Database Inputs (Using Free license)

0 Karma
Highlighted

Re: 500 Internal Server Error after upgrading to 6.4.0

Path Finder

We had the same problem. The mentioned workaround in Highlighted issues did't work for us.
Option 1 had no effect. Using option 2 returned an "bash: unset: `http://my-http-proxy:8080': not a valid identifier" error.
A slighlty modified version of option 2 worked for us:

echo $http{,s}_proxy
unset http_proxy
unset https_proxy

Instead of

echo $http{,s}_proxy 
unset $http{,s}_proxy

Maybe it also helps others.

Highlighted

Re: 500 Internal Server Error after upgrading to 6.4.0

Path Finder

You're correct, unset works with the variable name, not its value. Splunk ReleaseNotes page has a typo in their instructions.