Splunkweb won't start after applying 4.0.11

[dave_duvall]

So I have a lab box where I have applied 4.0.11 to my existing 4.0.10 installation.

Ran the upgrade using rpm -U and didn't have any issues with the upgrade. After applying the upgrade however splunkweb fails to start (splunkd starts fine)

Here is my web.conf from $SPLUNK_HOME/etc/system/local:

[settings]
enableSplunkWebSSL = 1
httpport = 443
mgmtHostPort = 127.0.0.1:8090

Here are the errors I see in splunkd.log:

05-06-2010 15:30:59.598 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem
05-06-2010 15:30:59.598 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
05-06-2010 15:31:03.204 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem
05-06-2010 15:31:03.204 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
05-06-2010 15:31:03.204 ERROR HTTPServer - SSL will not be enabled

Here is the directory list of /opt/splunk/etc/auth/server.pem:

[root@dctlallog03 local]# ls /opt/splunk/etc/auth/ -al
total 44
drwx------  4 splunk splunk 4096 May  6 15:03 .
drwxr-xr-x 16 splunk splunk 4096 May  6 15:11 ..
drwx--x--x  2 splunk splunk 4096 Apr 15 14:10 audit
-r--r--r--  1 splunk splunk  912 Apr 28 03:42 cacert.pem
-r--r--r--  1 splunk splunk 1875 Apr 28 03:42 ca.pem
-rw-r--r--  1 splunk splunk   17 Apr 15 14:10 ca.srl
drwx--x--x  2 splunk splunk 4096 Apr 15 14:10 distServerKeys
-rw-r--r--  1 splunk splunk  963 Apr 15 14:10 privkeySecure.pem
-rw-r--r--  1 splunk splunk  586 Apr 15 14:10 req.pem
-rw-r--r--  1 splunk splunk 2689 Apr 15 14:10 server.pem
-r--------  1 splunk splunk  255 Apr 15 14:10 splunk.secret

After upgrade I noticed that ca.srl, privkeySecure.pem, req.pem, and server.pem were is 600 mode so I tried a chmod 644 to see if that helped anything and it did not.

Please let me know if this is a bug in the install or if I somehow corrupted this key during upgrade.

Thanks,

Dave

[dave_duvall · Accepted solution]

So I figured out what this one was. Turns out somehow the upgrade to 4.0.11 must have modified server.conf. Since this was an install in my lab I was still using all of the out of the box certificates.

I edited /opt/splunk/etc/system/local/server.conf and provided the default SSL certificate password of "password" in sslKeysfilePassword entry.

Once I restarted splunk the password was re-encrypted in the config file and now things stop and start normally.

Does anyone know if the algorithm used to encrypt the password in that file was changed from 4.0.10 to 4.0.11? An upgrade shouldn't have modified that config.

Dave

[dave_duvall]

Official word from support:
This does happen in certain upgrade cases, we've been telling people to regenerate their server.conf if splunkweb is having startup issues. Renaming the file works as well as splunk will autogenerate a SSL password.