500 Internal Server Error

[aoleske]

Good morning,
I am trying to figure out an error 500 error on 6 new Centos 7 servers. Here is my scenario: Splunk Web comes up without an issue. I can connect to the URL and get a login screen. When I put the admin/password in, it immediately pops a "500 Internal Server Error" error screen. An interesting side note is that if I put the initial password in wrong it keeps prompting me for the correct password, so Splunk appears to be resolving the password. I have tried changing the password using " splunk edit user admin -password newPassowrd -auth admin:changeme" and also the method of using user-seed.conf to no avail. I am getting the following error in the web_service.log:

2019-11-07 14:47:28,051 INFO [5dc490f00a7f11b0320f50] error:323 - GET /en-US/ 127.0.0.1 8065
2019-11-07 14:47:28,051 INFO [5dc490f00a7f11b0320f50] error:324 - 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request.
2019-11-07 14:47:28,051 ERROR [5dc490f00a7f11b0320f50] error:325 - Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/cherrypy/cprequest.py", line 606, in respond
cherrypy.response.body = self.handler()
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/htmlinjectiontoolfactory.py", line 72, in wrapper
resp = handler(*args, **kwargs)
File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 25, in __call_
return self.callable(*self.args, **self.kwargs)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 38, in rundecs
return fn(*a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 119, in check
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 167, in validate_ip
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 246, in preform_sso_check
update_session_user(sessionKey, remote_user)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 189, in update_session_user
en = splunk.entity.getEntity('authentication/users', user, sessionKey=sessionKey)
File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 265, in getEntity
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py", line 500, in simpleRequest
raise splunk.SplunkdConnectionException('Error connecting to %s: %s' % (path, str(e)))
SplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /services/authentication/users/admin: [Errno 111] Connection refused',)

My splunk-launch.conf file:

#   Version 7.3.2

# Modify the following line to suit the location of your Splunk install.
# If unset, Splunk will use the parent of the directory containing the splunk
# CLI executable.
#
 SPLUNK_HOME=/opt/splunk

# By default, Splunk stores its indexes under SPLUNK_HOME in the
# var/lib/splunk subdirectory.  This can be overridden
# here:
#
# SPLUNK_DB=/opt/splunk-home/var/lib/splunk
SPLUNK_DB=/indexes/splunk
# Splunkd daemon name
SPLUNK_SERVER_NAME=splunkd

# Splunkweb daemon name
SPLUNK_WEB_NAME=splunkweb

# Changing the default bind from 127.0.0.1 to actual IP
SPLUNK_BINDIP=10.72.10.XXX

# If SPLUNK_OS_USER is set, then Splunk service will only start
# if the 'splunk [re]start [splunkd]' command is invoked by a user who
# is, or can effectively become via setuid(2), $SPLUNK_OS_USER.
# (This setting can be specified as username or as UID.)
#
 SPLUNK_OS_USER=splunk

My web.conf file:

# location of splunkd; don't include http[s]:// in this anymore.
mgmtHostPort = 10.72.10.XXX:8089

NOTE:
I tried adding "server.socket_host = 10.72.10.106" as one Answers response suggested for a similar problem - it has no effect.

My server.conf:

[general]
serverName = splunk-idx1.FQDN.com
pass4SymmKey = $7$qKGYODrBHTX/iU48JAtICIiYzLSTkJ/cJV0WReqxkB==

[sslConfig]
sslPassword = $7$yq5VWVVe1wXyurxFgpnBOQZx8XjykeVyg4xmW4Q9KOK8==

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

It seems like it is having an authentication issue within Splunk itself, but I am not sure where to look at this point:
SplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /services/authentication/users/admin: [Errno 111] Connection refused',)

Any help would be greatly appreciated!

[aoleske · Accepted solution]

My web.conf file had an error - needed to add [settings]:

# location of splunkd; don't include http[s]:// in this anymore.
mgmtHostPort = 10.72.10.XXX:8089

The correct config is:

# location of splunkd; don't include http[s]:// in this anymore.
[settings]
mgmtHostPort = 10.72.10.XXX:8089

[aoleske]

It looks like the certificate is fine, and I can make a connection with curl:

[user@splunk-XXX network-scripts]$ curl -k --user admin:XXXXX --trace-ascii dump.txt https://splunk-XXX:8089/services/authentication/users
== Info: About to connect() to splunk-XXX port 8089 (#0)
== Info: Trying 10.72.10.XXX...
== Info: Connected to splunk-XXX (10.72.10.XXX) port 8089 (#0)
== Info: Initializing NSS with certpath: sql:/etc/pki/nssdb
== Info: skipping SSL peer certificate verification
== Info: SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
== Info: Server certificate:
== Info: subject: O=SplunkUser,CN=SplunkServerDefaultCert
== Info: start date: Nov 08 17:11:36 2019 GMT
== Info: expire date: Nov 07 17:11:36 2022 GMT
== Info: common name: SplunkServerDefaultCert
== Info: issuer: E=support@splunk.com,CN=SplunkCommonCA,O=Splunk,L=San Francisco,ST=CA,C=US
== Info: Server auth using Basic with user 'admin'
=> Send header, 156 bytes (0x9c)
0000: GET /services/authentication/users HTTP/1.1
002d: Authorization: Basic YWRtaW46c3BsdW5rMjAxOA==
005c: User-Agent: curl/7.29.0
0075: Host: splunk-XXX:8089
008d: Accept: /

[fernanlee]

Probably you installed a universal forwarder in the same server.

Could you please run the following command in your CentOS server:

$ ps -A | grep splunk

And show me the output please?

[aoleske]

Your HW is compliant with the minimun and recommended specifications? Yes
Your server and OS is 64bit? Yes (CentOS 7)
Do you have enough space in your server? Yes
If the HW is not enough you will have a lot of problems.
Please check the following log:
$ cat SPLUNK_HOME/var/log/splunk/health.log: All green
Did you install splunk with a noon root user? Installed as splunk user
$ ls -la $SPLUNK_HOME: /opt/splunk
Remember DIAG is your best FRIEND!
Diag provides insight into your instance:
- How is the isntance configured
- What was the condition up the point that diag ran
Gather data based on Splunk components:
- OS setting, internal lgos, configuration files, etc.
- Produces a tar.gz file and diag.log
No customer data is retrieved.
$./splunk diag
-> Check for any errors.
Did you make an Splunk upgrade? No Upgrade - fresh install on new VM
Check the logs to verify if there was a problem during the process: No problems reported
SPLUNK_HOME/var/log/splunk/migration.log. N/A
= Notable Logs for crashes =
Check if crash*log exists or splunkd_stderr.log: Nothing interesting

[fernanlee]

Splunk diag?

[aoleske]

Not sure what you are looking for with splunk diag? It runs without errors. I could pull specific files out if needed and post the results here. I have also opened a ticket with support and uploaded the diag file there.

[aoleske]

splunk-7.3.2-c60db69f8e32-Linux-x86_64.tgz Is actually what we are currently running. I tried splunk-7.2.0-8c86330ac18-Linux-x86_64.tgz also to see if it made a difference, since we are running it successfully on a test server. splunk7.3.2 is now the only install currently on the box. I have 6 servers all with the same issue.

[aoleske]

thanks for the response!
No forwarder installed... This is a new install and the only thing installed is Splunk Enterprise:
splunk-7.2.0-8c86330ac18-Linux-x86_64.tgz

$ ps -A | grep splunk
9515 ? 00:00:11 splunkd
9517 ? 00:00:00 splunkd
9732 ? 00:00:01 splunkd

[fernanlee]

In this kind of errors we must perform a check list to ensure everything was deploy properly.

Your HW is compliant with the minimun and recommended specifications?
Your server and OS is 64bit?
Do you have enough space in your server?
If the HW is not enough you will have a lot of problems.

Please check the following log:
$ cat SPLUNK_HOME/var/log/splunk/health.log

Did you install splunk with a noon root user?
$ ls -la $SPLUNK_HOME

Remember DIAG is your best FRIEND!

Diag provides insight into your instance:
- How is the isntance configured
- What was the condition up the point that diag ran
Gather data based on Splunk components:
- OS setting, internal lgos, configuration files, etc.
- Produces a tar.gz file and diag.log

No customer data is retrieved.

$./splunk diag
-> Check for any errors.

Did you make an Splunk upgrade?

Check the logs to verify if there was a problem during the process:
SPLUNK_HOME/var/log/splunk/migration.log.

= Notable Logs for crashes =
Check if crash*log exists or splunkd_stderr.log

Hope this help!