Why do I get HTTP 500 Internal Server Error occurs when logging on Splunkweb UI ?

[sdubey_splunk]

splunkweb is slow. When we open browser -> enters https://x.x.x.x:8000 It takes time for login page to be visible. Once we enter the user/name password after few minutes we see get error message : 500 Internal Server Error.

Errors as per web_access.log

127.0.0.1 - - [30/Jan/2019:16:43:40.955 +0800] "GET /en-US/config HTTP/1.1" 200 288 "http://x.x.x.x:8000/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C)" - 5c5163bcf48e0ed51940 30004ms
127.0.0.1 - - [30/Jan/2019:16:44:11.234 +0800] "GET /en-US/static/@000/fonts/splunkicons-regular-webfont.woff HTTP/1.1" 304 - "http://x.x.x.x:8000/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C)" - 5c5163db3b8e0ed51940 3ms
127.0.0.1 - - [30/Jan/2019:16:43:55.110 +0800] "GET /en-US/config HTTP/1.1" 200 288 "http://x.x.x.x:8000/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C)" - 5c5163cb1c8e0ed51c18 30021ms
127.0.0.1 - - [30/Jan/2019:16:44:29.395 +0800] "GET /en-US/static/@89C48690B44183ACC5D6E663F8426C6814059E0AFB64E6F0ED33C5D83839FAE5/img/bookmark/ms-tileicon-144x144.png HTTP/1.1" 200 3170 "" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; Win64; x64; Trident/7.0; .NET4.0E; .NET4.0C)" - 5c5163ed658e0ed51198 5ms
127.0.0.1 - - [30/Jan/2019:16:44:17.857 +0800] "GET /en-US/ HTTP/1.1" 500 3047 "http://x.x.x.x:8000/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C)" - 5c5163e1db8e0ed51fd0 60017ms

What did we do to fix it ?
1. As we noticed that 60017ms ie more than 60 secs and default splunkdConnectionTimeout is 30 secs we tried changing to 180 seconds and it did not fix the issue.

2.started splunk with -debug option to collect debug errors/messages. It was not of much help same errors in web_access.log.

1. We made sure the default management port is not disabled. (https://answers.splunk.com/answers/170065/why-am-i-getting-the-following-error-logging-into.html)

To brute force a quick-fix until you sort out your configuration files, just put this in /opt/splunk/etc/system/local/server.conf:

[httpServer]
disableDefaultPort = false
Then restart Splunk.

1. Checked load on server : cpu/memory usage is very low. there is no load on server.

2. Exhausted all the options

[sdubey_splunk]

I tested in my lab and found that you get "Error connecting: SSL not configured on client" if you sslPassword is incorrect in server.conf.

Lab testing details
1. One server as setup License master
2. One server as license slave configuration was working fine and license was able to connect License master node.
3. On license slave : I took a copy of server.conf and changed sslPassword to incorrect one.
4. After changing the sslPassword I started seeing below errors in splunkd.log ( similar errors noticed on your server)
../../../var/log/splunk/splunkd.log:02-27-2019 06:14:10.261 +0000 ERROR LMTracker - failed to ping master=https://y.y.y.y:8089, check server.conf for master_uri setting, this may lead to search being disabled for this machine err='Unable to connect to license master=https://y.y.y.y:8089 Error connecting: SSL not configured on client'
../../../var/log/splunk/splunkd.log:02-27-2019 06:14:11.240 +0000 ERROR LMTracker - failed to send rows, reason='Unable to connect to license master=https://y.y.y.y:8089 Error connecting: SSL not configured on client'
../../../var/log/splunk/splunkd.log:02-27-2019 06:15:11.241 +0000 ERROR LMTracker - failed to send rows, reason='Unable to connect to license master=https://y.y.y.y:8089 Error connecting: SSL not configured on client'
5. And similar errors noticed while checking the licensing details( Settings-> license details).

Solution : . Make sure you have the correct sslPassword and update in $SPLU
K_HOME/etc/system/local/server.conf