500 Internal Server Error after upgrading from 6.5.3 to 7.1.3.

[shd]

Today I have upgraded from Splunk-6.3.3 to Splunk-6.4.0 and web interface has stopped working. I get HTTP 500 Internal Server Error each time I try to open any page of Splunk.

I am using SSO, but it doesn't work even after disabling it. CLI works fine, I can login an make a request there.

Log of clean try to login:

2016-04-06 13:33:31,465 ERROR   [5704c9db347f5e5c2cae50] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:31,467 INFO    [5704c9db347f5e5c2cae50] decorators:362 - require_login - no splunkd sessionKey variable set; cherrypy_session=f7881bf5cdb6e18c9d603e35261fb4d6eec6b9d1 request_path=/en-US/
2016-04-06 13:33:31,468 INFO    [5704c9db347f5e5c2cae50] decorators:383 - require_login - redirecting to login
2016-04-06 13:33:31,745 ERROR   [5704c9db827f5e5c321d90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:31,971 ERROR   [5704c9db827f5e5c321d90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:32,455 ERROR   [5704c9dc397f5e5c2caf90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:36,591 ERROR   [5704c9e0547f5e5c2ca390] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:36,817 ERROR   [5704c9e0547f5e5c2ca390] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:37,042 DEBUG   [5704c9e0547f5e5c2ca390] _cplogging:55 - [06/Apr/2016:13:33:37] HTTP Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 606, in respond
    cherrypy.response.body = self.handler()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 25, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 38, in rundecs
    return fn(*a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 118, in check
    return fn(self, *a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 167, in validate_ip
    return fn(self, *a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 407, in handle_exceptions
    return fn(self, *a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 462, in apply_cache_headers
    response = fn(self, *a, **kw)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py", line 176, in login
    sessionKey = splunk.auth.getSessionKey(username, password, hostPath=self.splunkd_urlhost, newPassword=newpassword)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/auth.py", line 31, in getSessionKey
    serverResponse, serverContent = rest.simpleRequest(uri, postargs=args)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 516, in simpleRequest
    raise splunk.AuthorizationFailed(extendedMessages=uri)
AuthorizationFailed: [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/auth/login

2016-04-06 13:33:37,042 INFO    [5704c9e0547f5e5c2ca390] _cplogging:55 - [06/Apr/2016:13:33:37] HTTP
Request Headers:
  USER-AGENT: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
  REFERER: https://my-noc-instance.domain.com/splunk/en-US/account/login?return_to=%2Fsplunk%2Fen-US%2F
  Content-Type: application/x-www-form-urlencoded
  X-FORWARDED-HOST: my-noc-instance.domain.com
  ACCEPT-LANGUAGE: en-US,en;q=0.5
  HOST: my-noc-instance.domain.com:8000
  ACCEPT-ENCODING: gzip, deflate
  X-FORWARDED-SERVER: my-noc-instance.domain.com
  ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  X-FORWARDED-FOR: 10.10.10.10
  CONNECTION: Keep-Alive
  COOKIE: cval=2069472843; session_id_8000=f7881bf5cdb6e18c9d603e35261fb4d6eec6b9d1; splunkweb_uid=57DDAC14-229E-4E10-80D8-F573359D9066
  DNT: 1
  X-REMOTE-USER: username@DOMAIN.COM
  Content-Length: 85
  VIA: 1.1 my-noc-instance.domain.com
  Remote-Addr: 127.0.0.1

[woodcock]

This happens when you accidentally turn off the management interface. You can look for this with either of these 2 commands:

find /opt/splunk/etc/ -type f -name server.conf -exec grep -il disableDefaultPort {} \;
/opt/splunk/bin/splunk btool server list --debug | grep disableDefaultPort

To brute force a quick-fix until you sort out your configuration files, just put this in /opt/splunk/etc/system/local/server.conf:

[httpServer]
disableDefaultPort = false

Then restart Splunk.
To add insult to injury, neither the splunk logs, nor the dead page served to you give you any indication that this is the situation and either could and BOTH SHOULD. Even when we turned on debug with /opt/splunk/bin/splunk start --debug, we STILL do not get any log telling us that this setting has explicitly disabled this core function. The ONLY place that you see this, and the only reason that we figured it out, is that it IS logged to STDOUT when you start splunk. You will see this somewhat casual note:

$ /opt/splunk/bin/splunk start

Splunk> All batbelt. No tights.

Checking prerequisites...
        Management port has been set disabled; the web UI cannot work.
        Checking http port [8000]: open
        Management port has been set disabled; cli support for this configuration is currently incomplete.

I opened a P4/ER to have this logged as a WARN but who knows if this will ever get implemented. Hopefully this answer will save somebody the day that I wasted on this. To be fair, it was my own fault; I was hardening UFs and did not have my blacklist correct for my server class so it hit a few of my Search Heads. DOH!

[nzarzyckivs]

You saved me a ton of troubleshooting hours by posting this. Thank you for taking the time to do it.

[mlorch]

We had the same problem. The mentioned workaround in Highlighted issues did't work for us.
Option 1 had no effect. Using option 2 returned an "bash: unset: `http://my-http-proxy:8080': not a valid identifier" error.
A slighlty modified version of option 2 worked for us:

echo $http{,s}_proxy
unset http_proxy
unset https_proxy

Instead of

echo $http{,s}_proxy 
unset $http{,s}_proxy

Maybe it also helps others.

[ontkanin]

You're correct, unset works with the variable name, not its value. Splunk ReleaseNotes page has a typo in their instructions.

[ChrisG]

Thanks for pointing that out, I have updated the original answer and the release notes.

[ChrisG · Accepted solution]

The release notes are updated with the following workaround information:

Option 1: Export the following setting into your environment prior to starting Splunk Enterprise: NO_PROXY="127.0.0.1,localhost,[::1]"

export NO_PROXY="127.0.0.1,localhost,[::1]"
or add it to $SPLUNK_HOME/etc/splunk-launch.conf and restart Splunk Enterprise.

Option 2: Unset the proxy on the Linux system and restart Splunk Enterprise:

`echo $http{,s}_proxy
unset http{,s}_proxy'

If the proxy is set in splunk-launch.conf but is not actually used or needed, remove the setting and restart.

[shd]

Works now. Thanks!

[Vijaikanth]

Hi,

In which file we need to put the below code

echo $http{,s}_proxy
unset http{,s}_proxy'

or where we need to execute this code?

[mlorch]

Yes, it's a Unix command. You have to execute it in the shell. Of course you can wrap it in shell script, the "./splunk start" at the end.
But why not take 6.4.1 or 6.4.2? This has only been an issue in 6.4.0.

[Vijaikanth]

Thanks. But this problem also happens in 6.4.2 where the only page affected is Database Inputs (Using Free license)

[ChrisG]

Corrected option 2 per the Answer and comment below.

[ChrisG]

There have been other reports of this. If you have an active support agreement, please open a case so Support can track the issues across customer sites.

We are developing workaround information and will update the release notes soon.