Security

Why am I getting the following error logging into Splunk? "500 Internal Server Error ResponseNotReady"

Could someone give me solution for the problem.

When i am trying to login into splunk showing below message in page.

*500 Internal Server Error
Return to Splunk home page
ResponseNotReady
View more information about your request (request ID = 54219c3173e66d493160) in Search

This page was linked to from https://myHost:8000/en-US/account/login?return_to=%2Fen-US%2F.

You are using myHost:8000, which is connected to splunkd @000 at https://127.0.0.1:8089 on Tue Sep 23 11:13:39 2014.*

Labels (1)
Tags (3)

I'm getting the same issue.
background:
I am automating a base deployment with ansible. The architecture consists of 3 clustered indexers, 3 clustered search heads, 1 cluster master, 1 deployment server, 1 license master. I was able to login without ssl configured. I then edited the web.conf file to enable web ssl and pointed it at the private key and server certificate. I then wiped the servers and re-ran the ansible scripts. I also have splunk running through systemd. I have set the permissions of /opt/splunk to :splunk and made the necessary additions to sudoers.

Result:
The web UI shows up at the set url registering a valid ssl certificate; however, once I utilize the user to login it fails with a 500 Internal Server Error

logs:
The webservice.log has the following entry:
"File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/
init_.py", line 500, in simpleRequest raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e)) SplunkdConnectionException: Splunkd Daemon is not responding: ("Error connecting to /services/authentication/users/: (')ssl.c:725: The handshake operation timed out',)",)"

0 Karma

New Member

I'm still getting a 500 error on login to the management interface at http://mypublicip:8000/

I've tried all the above suggestions but no joy.

This is running on CentOS 7.5.1804 as the user "splunk" with a home directory of /opt/splunk

Any other suggestions would be greatly appreciated.

,I'm having the same problem - setting:

[httpServer]
disableDefaultPort = false

in /opt/splunk/etc/system/local/server.conf does not fix this. Still getting a 500 error when logging into the admin page at http://mypublicip:8000

I've tried the other recommended settings - namely setting PATH and SPLUNKHOME vars and sourcing $SPLUNKHOME/bin/setSplunkEnv

Still no joy - any thoughts?

0 Karma

Engager

not sure why the above answer got down votes but web_service.log was the file where I see actual errors regarding the 500 error on the web UI.

0 Karma

Esteemed Legend

This happens when you accidentally turn off the management interface. You can look for this with either of these 2 commands:

find /opt/splunk/etc/ -type f -name server.conf -exec grep -il disableDefaultPort {} \;
/opt/splunk/bin/splunk btool server list --debug | grep disableDefaultPort

To brute force a quick-fix until you sort out your configuration files, just put this in /opt/splunk/etc/system/local/server.conf:

[httpServer]
disableDefaultPort = false

Then restart Splunk.
To add insult to injury, neither the splunk logs, nor the dead page served to you give you any indication that this is the situation and either could and BOTH SHOULD. Even when we turned on debug with /opt/splunk/bin/splunk start --debug, we STILL do not get any log telling us that this setting has explicitly disabled this core function. The ONLY place that you see this, and the only reason that we figured it out, is that it IS logged to STDOUT when you start splunk. You will see this somewhat casual note:

$ /opt/splunk/bin/splunk start

Splunk> All batbelt. No tights.

Checking prerequisites...
        Management port has been set disabled; the web UI cannot work.
        Checking http port [8000]: open
        Management port has been set disabled; cli support for this configuration is currently incomplete.

I opened a P4/ER to have this logged as a WARN but who knows if this will ever get implemented. Hopefully this answer will save somebody the day that I wasted on this. To be fair, it was my own fault; I was hardening UFs and did not have my blacklist correct for my server class so it hit a few of my Search Heads. DOH!

Splunk Employee
Splunk Employee

Thank you so much. This and rectifying permissions seemed to fix my issue.

New Member

We got this on a brand new 6.6.7 install on RHEL 6.9. The UI was responding very slowly then timing out and giving us the 500 error.

We reinstalled Splunk and evidently at first it was working fine. In the morning, it stopped responding again, so we stopped SSL. Things worked fine, so we turned SSL back on and everything has worked fine since.

0 Karma

Engager

Another way this error will occur is if the file ownership or permissions on your Splunk directories are changed.
I had an admin inadvertently move ownership of /splunk on my servers and it immediately started throwing these 500 errors to my splunk users.

A simple 'ls -lar' or splunk status will confirm. Once ownership was restored the 500 errors went away.
Some search and alerting functions did require a restart of splunk to fix.

Splunk Employee
Splunk Employee

Thank you. This fixed my issue.

Engager

It seems to be happening due to not unsetting proxy values as "akhtet" said.
I used a below generic bash script on my mac and it fixed the problem.

==splunk.sh==

#!/bin/bash
if [ "$1" = "start" ] ; then
export SPLUNK_HOME=/Applications/Splunk/
source $SPLUNK_HOME/bin/setSplunkEnv
export PATH=$SPLUNK_HOME/bin:$PATH
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY
fi
/Applications/Splunk/bin/splunk $1

usage :
./splunk.sh start
./splunk.sh stop
./splunk.sh status
./splunk.sh

0 Karma

New Member

You may also want to check your proxy settings.

I had the same error message as soon as I login to Splunk.
When I looked into web_service.log, I saw the following error message.

startup:96 - Unable to read in product version information; (403, 'Forbidden')

The root cause is that I have set the http(s)proxy environment variable (as I'm inside a proxied environment) and Splunk seem to have issues connecting to that proxy to read in product version info. After I have reset the http(s)proxy variable to empty strings, the issue no longer exists.

0 Karma

Splunk Employee
Splunk Employee

Be sure to set the following before starting for the first time:

export SPLUNKHOME=/my/splunk/dir
source $SPLUNK
HOME/bin/setSplunkEnv
Add $SPLUNKHOME/bin to your PATH
export PATH=$SPLUNK
HOME/bin:$PATH

Check to make sure 'which python' is referencing the local, splunk, python libs:

[splunkysploo]# which python
/<$SPLUNK_HOME>/bin/python

Then, start Splunk with $SPLUNK_HOME/bin/splunk start as per usual.

Communicator

This is a very generic error message. The first place I would look would be /splunk/var/log/splunk/web_service.log for further details.