Installation

Splunkweb won't start after applying 4.0.11

dave_duvall
Explorer

So I have a lab box where I have applied 4.0.11 to my existing 4.0.10 installation.

Ran the upgrade using rpm -U and didn't have any issues with the upgrade. After applying the upgrade however splunkweb fails to start (splunkd starts fine)

Here is my web.conf from $SPLUNK_HOME/etc/system/local:

[settings]
enableSplunkWebSSL = 1
httpport = 443
mgmtHostPort = 127.0.0.1:8090

Here are the errors I see in splunkd.log:

05-06-2010 15:30:59.598 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem
05-06-2010 15:30:59.598 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
05-06-2010 15:31:03.204 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem
05-06-2010 15:31:03.204 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
05-06-2010 15:31:03.204 ERROR HTTPServer - SSL will not be enabled

Here is the directory list of /opt/splunk/etc/auth/server.pem:

[root@dctlallog03 local]# ls /opt/splunk/etc/auth/ -al
total 44
drwx------  4 splunk splunk 4096 May  6 15:03 .
drwxr-xr-x 16 splunk splunk 4096 May  6 15:11 ..
drwx--x--x  2 splunk splunk 4096 Apr 15 14:10 audit
-r--r--r--  1 splunk splunk  912 Apr 28 03:42 cacert.pem
-r--r--r--  1 splunk splunk 1875 Apr 28 03:42 ca.pem
-rw-r--r--  1 splunk splunk   17 Apr 15 14:10 ca.srl
drwx--x--x  2 splunk splunk 4096 Apr 15 14:10 distServerKeys
-rw-r--r--  1 splunk splunk  963 Apr 15 14:10 privkeySecure.pem
-rw-r--r--  1 splunk splunk  586 Apr 15 14:10 req.pem
-rw-r--r--  1 splunk splunk 2689 Apr 15 14:10 server.pem
-r--------  1 splunk splunk  255 Apr 15 14:10 splunk.secret

After upgrade I noticed that ca.srl, privkeySecure.pem, req.pem, and server.pem were is 600 mode so I tried a chmod 644 to see if that helped anything and it did not.

Please let me know if this is a bug in the install or if I somehow corrupted this key during upgrade.

Thanks,

Dave

1 Solution

dave_duvall
Explorer

So I figured out what this one was. Turns out somehow the upgrade to 4.0.11 must have modified server.conf. Since this was an install in my lab I was still using all of the out of the box certificates.

I edited /opt/splunk/etc/system/local/server.conf and provided the default SSL certificate password of "password" in sslKeysfilePassword entry.

Once I restarted splunk the password was re-encrypted in the config file and now things stop and start normally.

Does anyone know if the algorithm used to encrypt the password in that file was changed from 4.0.10 to 4.0.11? An upgrade shouldn't have modified that config.

Dave

View solution in original post

dave_duvall
Explorer

So I figured out what this one was. Turns out somehow the upgrade to 4.0.11 must have modified server.conf. Since this was an install in my lab I was still using all of the out of the box certificates.

I edited /opt/splunk/etc/system/local/server.conf and provided the default SSL certificate password of "password" in sslKeysfilePassword entry.

Once I restarted splunk the password was re-encrypted in the config file and now things stop and start normally.

Does anyone know if the algorithm used to encrypt the password in that file was changed from 4.0.10 to 4.0.11? An upgrade shouldn't have modified that config.

Dave

dave_duvall
Explorer

Official word from support:
This does happen in certain upgrade cases, we've been telling people to regenerate their server.conf if splunkweb is having startup issues. Renaming the file works as well as splunk will autogenerate a SSL password.

Get Updates on the Splunk Community!

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...