Security

Why am I getting the following error logging into Splunk? "500 Internal Server Error ResponseNotReady"

nagaraju_varkal
Engager

Could someone give me solution for the problem.

When i am trying to login into splunk showing below message in page.

*500 Internal Server Error
Return to Splunk home page
ResponseNotReady
View more information about your request (request ID = 54219c3173e66d493160) in Search

This page was linked to from https://myHost:8000/en-US/account/login?return_to=%2Fen-US%2F.

You are using myHost:8000, which is connected to splunkd @000 at https://127.0.0.1:8089 on Tue Sep 23 11:13:39 2014.*

Labels (1)
Tags (3)

dips1213
New Member

Thanks for all the info here. I had a similar issue in our lab env. Figured out it was due to mgmt port being used by some other process. 
Splunk start command showed me the error and also allowed me to change it. 

dips1213_0-1618840757054.png

 

0 Karma

jingisonsplunk
Engager

I'm getting the same issue.
background:
I am automating a base deployment with ansible. The architecture consists of 3 clustered indexers, 3 clustered search heads, 1 cluster master, 1 deployment server, 1 license master. I was able to login without ssl configured. I then edited the web.conf file to enable web ssl and pointed it at the private key and server certificate. I then wiped the servers and re-ran the ansible scripts. I also have splunk running through systemd. I have set the permissions of /opt/splunk to :splunk and made the necessary additions to sudoers.

Result:
The web UI shows up at the set url registering a valid ssl certificate; however, once I utilize the user to login it fails with a 500 Internal Server Error

logs:
The web_service.log has the following entry:
"File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py", line 500, in simpleRequest raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e)) SplunkdConnectionException: Splunkd Daemon is not responding: ("Error connecting to /services/authentication/users/: (')ssl.c:725: The handshake operation timed out',)",)"

0 Karma

BestWestern
Loves-to-Learn

@jingisonsplunk Did you find a solution to this? 

0 Karma

gavstah
New Member

I'm still getting a 500 error on login to the management interface at http://mypublicip:8000/

I've tried all the above suggestions but no joy.

This is running on CentOS 7.5.1804 as the user "splunk" with a home directory of /opt/splunk

Any other suggestions would be greatly appreciated.

,I'm having the same problem - setting:

[httpServer]
disableDefaultPort = false

in /opt/splunk/etc/system/local/server.conf does not fix this. Still getting a 500 error when logging into the admin page at http://mypublicip:8000

I've tried the other recommended settings - namely setting PATH and SPLUNK_HOME vars and sourcing $SPLUNK_HOME/bin/setSplunkEnv

Still no joy - any thoughts?

0 Karma

jheatley
Engager

not sure why the above answer got down votes but web_service.log was the file where I see actual errors regarding the 500 error on the web UI.

0 Karma

woodcock
Esteemed Legend

This happens when you accidentally turn off the management interface. You can look for this with either of these 2 commands:

find /opt/splunk/etc/ -type f -name server.conf -exec grep -il disableDefaultPort {} \;
/opt/splunk/bin/splunk btool server list --debug | grep disableDefaultPort

To brute force a quick-fix until you sort out your configuration files, just put this in /opt/splunk/etc/system/local/server.conf:

[httpServer]
disableDefaultPort = false

Then restart Splunk.
To add insult to injury, neither the splunk logs, nor the dead page served to you give you any indication that this is the situation and either could and BOTH SHOULD. Even when we turned on debug with /opt/splunk/bin/splunk start --debug, we STILL do not get any log telling us that this setting has explicitly disabled this core function. The ONLY place that you see this, and the only reason that we figured it out, is that it IS logged to STDOUT when you start splunk. You will see this somewhat casual note:

$ /opt/splunk/bin/splunk start

Splunk> All batbelt. No tights.

Checking prerequisites...
        Management port has been set disabled; the web UI cannot work.
        Checking http port [8000]: open
        Management port has been set disabled; cli support for this configuration is currently incomplete.

I opened a P4/ER to have this logged as a WARN but who knows if this will ever get implemented. Hopefully this answer will save somebody the day that I wasted on this. To be fair, it was my own fault; I was hardening UFs and did not have my blacklist correct for my server class so it hit a few of my Search Heads. DOH!

Labbemiche
Engager

Same problem here. UF configs were applied to our HF. Still a good answer 4yrs later. Thank you.

Tags (1)
0 Karma

kkrishnan_splun
Splunk Employee
Splunk Employee

Thank you so much. This and rectifying permissions seemed to fix my issue.

swhipkey
New Member

We got this on a brand new 6.6.7 install on RHEL 6.9. The UI was responding very slowly then timing out and giving us the 500 error.

We reinstalled Splunk and evidently at first it was working fine. In the morning, it stopped responding again, so we stopped SSL. Things worked fine, so we turned SSL back on and everything has worked fine since.

0 Karma

dkretzmer
Explorer

Another way this error will occur is if the file ownership or permissions on your Splunk directories are changed.
I had an admin inadvertently move ownership of /splunk on my servers and it immediately started throwing these 500 errors to my splunk users.

A simple 'ls -lar' or splunk status will confirm. Once ownership was restored the 500 errors went away.
Some search and alerting functions did require a restart of splunk to fix.

kkrishnan_splun
Splunk Employee
Splunk Employee

Thank you. This fixed my issue.

kosarajh
Engager

It seems to be happening due to not unsetting proxy values as "akhtet" said.
I used a below generic bash script on my mac and it fixed the problem.

==splunk.sh==

#!/bin/bash
if [ "$1" = "start" ] ; then
export SPLUNK_HOME=/Applications/Splunk/
source $SPLUNK_HOME/bin/setSplunkEnv
export PATH=$SPLUNK_HOME/bin:$PATH
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY
fi
/Applications/Splunk/bin/splunk $1

usage :
./splunk.sh start
./splunk.sh stop
./splunk.sh status
./splunk.sh

0 Karma

akhtet
New Member

You may also want to check your proxy settings.

I had the same error message as soon as I login to Splunk.
When I looked into web_service.log, I saw the following error message.

startup:96 - Unable to read in product version information; (403, 'Forbidden')

The root cause is that I have set the http(s)_proxy environment variable (as I'm inside a proxied environment) and Splunk seem to have issues connecting to that proxy to read in product version info. After I have reset the http(s)_proxy variable to empty strings, the issue no longer exists.

0 Karma

weeb
Splunk Employee
Splunk Employee

Be sure to set the following before starting for the first time:

export SPLUNK_HOME=/my/splunk/dir
source $SPLUNK_HOME/bin/setSplunkEnv
Add $SPLUNK_HOME/bin to your PATH
export PATH=$SPLUNK_HOME/bin:$PATH

Check to make sure 'which python' is referencing the local, splunk, python libs:

[splunkysploo]# which python
/<$SPLUNK_HOME>/bin/python

Then, start Splunk with $SPLUNK_HOME/bin/splunk start as per usual.

Jeff_Lightly_Sp
Communicator

This is a very generic error message. The first place I would look would be /splunk/var/log/splunk/web_service.log for further details.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...