Installation

500 Internal Server Error after upgrading from 6.5.3 to 7.1.3.

shd
Explorer

Today I have upgraded from Splunk-6.3.3 to Splunk-6.4.0 and web interface has stopped working. I get HTTP 500 Internal Server Error each time I try to open any page of Splunk.

I am using SSO, but it doesn't work even after disabling it. CLI works fine, I can login an make a request there.

Log of clean try to login:

2016-04-06 13:33:31,465 ERROR   [5704c9db347f5e5c2cae50] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:31,467 INFO    [5704c9db347f5e5c2cae50] decorators:362 - require_login - no splunkd sessionKey variable set; cherrypy_session=f7881bf5cdb6e18c9d603e35261fb4d6eec6b9d1 request_path=/en-US/
2016-04-06 13:33:31,468 INFO    [5704c9db347f5e5c2cae50] decorators:383 - require_login - redirecting to login
2016-04-06 13:33:31,745 ERROR   [5704c9db827f5e5c321d90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:31,971 ERROR   [5704c9db827f5e5c321d90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:32,455 ERROR   [5704c9dc397f5e5c2caf90] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:36,591 ERROR   [5704c9e0547f5e5c2ca390] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:36,817 ERROR   [5704c9e0547f5e5c2ca390] startup:96 - Unable to read in product version information; [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/server/info
2016-04-06 13:33:37,042 DEBUG   [5704c9e0547f5e5c2ca390] _cplogging:55 - [06/Apr/2016:13:33:37] HTTP Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 606, in respond
    cherrypy.response.body = self.handler()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 25, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 38, in rundecs
    return fn(*a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 118, in check
    return fn(self, *a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 167, in validate_ip
    return fn(self, *a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 407, in handle_exceptions
    return fn(self, *a, **kw)
  File "<string>", line 1, in <lambda>
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 462, in apply_cache_headers
    response = fn(self, *a, **kw)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py", line 176, in login
    sessionKey = splunk.auth.getSessionKey(username, password, hostPath=self.splunkd_urlhost, newPassword=newpassword)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/auth.py", line 31, in getSessionKey
    serverResponse, serverContent = rest.simpleRequest(uri, postargs=args)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 516, in simpleRequest
    raise splunk.AuthorizationFailed(extendedMessages=uri)
AuthorizationFailed: [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/services/auth/login

2016-04-06 13:33:37,042 INFO    [5704c9e0547f5e5c2ca390] _cplogging:55 - [06/Apr/2016:13:33:37] HTTP
Request Headers:
  USER-AGENT: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
  REFERER: https://my-noc-instance.domain.com/splunk/en-US/account/login?return_to=%2Fsplunk%2Fen-US%2F
  Content-Type: application/x-www-form-urlencoded
  X-FORWARDED-HOST: my-noc-instance.domain.com
  ACCEPT-LANGUAGE: en-US,en;q=0.5
  HOST: my-noc-instance.domain.com:8000
  ACCEPT-ENCODING: gzip, deflate
  X-FORWARDED-SERVER: my-noc-instance.domain.com
  ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  X-FORWARDED-FOR: 10.10.10.10
  CONNECTION: Keep-Alive
  COOKIE: cval=2069472843; session_id_8000=f7881bf5cdb6e18c9d603e35261fb4d6eec6b9d1; splunkweb_uid=57DDAC14-229E-4E10-80D8-F573359D9066
  DNT: 1
  X-REMOTE-USER: username@DOMAIN.COM
  Content-Length: 85
  VIA: 1.1 my-noc-instance.domain.com
  Remote-Addr: 127.0.0.1

 

Labels (1)
1 Solution

ChrisG
Splunk Employee
Splunk Employee

The release notes are updated with the following workaround information:

Option 1: Export the following setting into your environment prior to starting Splunk Enterprise: NO_PROXY="127.0.0.1,localhost,[::1]"

export NO_PROXY="127.0.0.1,localhost,[::1]"
or add it to $SPLUNK_HOME/etc/splunk-launch.conf and restart Splunk Enterprise.

Option 2: Unset the proxy on the Linux system and restart Splunk Enterprise:

`echo $http{,s}_proxy
unset http{,s}_proxy'

If the proxy is set in splunk-launch.conf but is not actually used or needed, remove the setting and restart.

View solution in original post

woodcock
Esteemed Legend

This happens when you accidentally turn off the management interface. You can look for this with either of these 2 commands:

find /opt/splunk/etc/ -type f -name server.conf -exec grep -il disableDefaultPort {} \;
/opt/splunk/bin/splunk btool server list --debug | grep disableDefaultPort

To brute force a quick-fix until you sort out your configuration files, just put this in /opt/splunk/etc/system/local/server.conf:

[httpServer]
disableDefaultPort = false

Then restart Splunk.
To add insult to injury, neither the splunk logs, nor the dead page served to you give you any indication that this is the situation and either could and BOTH SHOULD. Even when we turned on debug with /opt/splunk/bin/splunk start --debug, we STILL do not get any log telling us that this setting has explicitly disabled this core function. The ONLY place that you see this, and the only reason that we figured it out, is that it IS logged to STDOUT when you start splunk. You will see this somewhat casual note:

$ /opt/splunk/bin/splunk start

Splunk> All batbelt. No tights.

Checking prerequisites...
        Management port has been set disabled; the web UI cannot work.
        Checking http port [8000]: open
        Management port has been set disabled; cli support for this configuration is currently incomplete.

I opened a P4/ER to have this logged as a WARN but who knows if this will ever get implemented. Hopefully this answer will save somebody the day that I wasted on this. To be fair, it was my own fault; I was hardening UFs and did not have my blacklist correct for my server class so it hit a few of my Search Heads. DOH!

nzarzyckivs
Explorer

You saved me a ton of troubleshooting hours by posting this. Thank you for taking the time to do it.

0 Karma

mlorch
Path Finder

We had the same problem. The mentioned workaround in Highlighted issues did't work for us.
Option 1 had no effect. Using option 2 returned an "bash: unset: `http://my-http-proxy:8080': not a valid identifier" error.
A slighlty modified version of option 2 worked for us:

echo $http{,s}_proxy
unset http_proxy
unset https_proxy

Instead of

echo $http{,s}_proxy 
unset $http{,s}_proxy

Maybe it also helps others.

ontkanin
Path Finder

You're correct, unset works with the variable name, not its value. Splunk ReleaseNotes page has a typo in their instructions.

ChrisG
Splunk Employee
Splunk Employee

Thanks for pointing that out, I have updated the original answer and the release notes.

0 Karma

ChrisG
Splunk Employee
Splunk Employee

The release notes are updated with the following workaround information:

Option 1: Export the following setting into your environment prior to starting Splunk Enterprise: NO_PROXY="127.0.0.1,localhost,[::1]"

export NO_PROXY="127.0.0.1,localhost,[::1]"
or add it to $SPLUNK_HOME/etc/splunk-launch.conf and restart Splunk Enterprise.

Option 2: Unset the proxy on the Linux system and restart Splunk Enterprise:

`echo $http{,s}_proxy
unset http{,s}_proxy'

If the proxy is set in splunk-launch.conf but is not actually used or needed, remove the setting and restart.

shd
Explorer

Works now. Thanks!

0 Karma

Vijaikanth
Path Finder

Hi,

In which file we need to put the below code

echo $http{,s}_proxy
unset http{,s}_proxy'

or where we need to execute this code?

0 Karma

mlorch
Path Finder

Yes, it's a Unix command. You have to execute it in the shell. Of course you can wrap it in shell script, the "./splunk start" at the end.
But why not take 6.4.1 or 6.4.2? This has only been an issue in 6.4.0.

0 Karma

Vijaikanth
Path Finder

Thanks. But this problem also happens in 6.4.2 where the only page affected is Database Inputs (Using Free license)

0 Karma

ChrisG
Splunk Employee
Splunk Employee

Corrected option 2 per the Answer and comment below.

0 Karma

ChrisG
Splunk Employee
Splunk Employee

There have been other reports of this. If you have an active support agreement, please open a case so Support can track the issues across customer sites.

We are developing workaround information and will update the release notes soon.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...