ERROR MongodRunner [9060 KVStoreConfigurationThread] - Failed to convert PEM to PFX

[jeremyhagand61]

Just putting this here for others who come across this problem since I got no results when I searched here.

After upgrading to Splunk 9.0.1 and configuring an SSL cert for the kvstore I got this error on one of my two instances. On Windows, the kvstore relies on the server cert being in the Windows local machine certificate store. At startup it converts the supplied PEM (with embedded cert and password protected key) into a PFX which it then imports into the store.

The error referenced in Subject is preceded by:

ERROR MongodRunner [9060 KVStoreConfigurationThread] - Command cmd="{CMD.EXE /C ( "C:\Program Files\Splunk\bin\openssl.exe" pkcs12 -inkey "C:\Program Files\Splunk\etc\auth\mycerts\splunkd.pem" -in "C:\Program Files\Splunk\etc\auth\mycerts\splunkd.pem" -passin pass:xxxx -export -out "C:\Program Files\Splunk\etc\auth\mycerts\splunkd.pem.pfx" -passout pass:xxxx )}" failed: exited with code 1. unable to load private key\r\n10460:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:.\crypto\evp\evp_enc.c:590:\r\n10460:error:0906A065:PEM routines:PEM_do_header:bad decrypt:.\crypto\pem\pem_lib.c:476:\r\n 

I tried this command and it did not work. I tried the openssl command by itself in PowerShell and it DID work. The problem turned out to be one of the special character in the password I had set on my private key. I used open SSL to write out a new copy of the key with a different password with no special characters and lo and behold it worked. Just be careful to replace the new password in all locations it might be used (EG: Twice in server.conf and in inputs.conf on an indexer).

The command for changing the password on your key is as follows:

.\openssl.exe rsa -aes265 -in mykey.key -out mynewkey.key -passin pass:oldpassword -passout pass:newpassword

Just make sure you are aware of what the special character in your old password might be and use PowerShell not the command prompt

[bseppanen1]

Just curious what special character you think that may have triggered this?  I'm seeing something possibly similar, but I'm skeptical it would be barfing on a "+"

This is with version 9.0.1, and is what I'm seeing

10-20-2022 15:23:30.702 -0400 INFO loader [5164 MainThread] - JsonWebToken Manager registration with KVStore successful.
10-20-2022 15:23:30.826 -0400 INFO MongodRunner [7720 KVStoreConfigurationThread] - Found an existing PFX certificate
10-20-2022 15:23:30.826 -0400 INFO MongodRunner [7720 KVStoreConfigurationThread] - Using mongod command line --sslCertificateSelector subject=US
10-20-2022 15:23:30.826 -0400 INFO MongodRunner [7720 KVStoreConfigurationThread] - Using mongod command line --tlsDisabledProtocols noTLS1_0,noTLS1_1
10-20-2022 15:23:30.826 -0400 INFO MongodRunner [7720 KVStoreConfigurationThread] - Using mongod command line --sslCipherConfig ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256
10-20-2022 15:23:30.826 -0400 INFO MongodRunner [7720 KVStoreConfigurationThread] - Using mongod command line --noscripting
10-20-2022 15:23:30.873 -0400 ERROR MongodRunner [9524 MongodLogThread] - mongod exited abnormally (exit code 1, status: exited with code 1) - look at mongod.log to investigate.
10-20-2022 15:23:30.873 -0400 ERROR KVStoreBulletinBoardManager [9524 MongodLogThread] - KV Store process terminated abnormally (exit code 1, status exited with code 1). See mongod.log and splunkd.log for details.
10-20-2022 15:23:30.873 -0400 WARN KVStoreConfigurationProvider [9524 MongodLogThread] - Action scheduled, but event loop is not ready yet
10-20-2022 15:23:30.873 -0400 ERROR KVStoreBulletinBoardManager [9524 MongodLogThread] - KV Store changed status to failed. KVStore process terminated..

[janderson42]

@bseppanen1 Mine barfed on a "^".

[MaxG]

I have a similar issue. Splunk can convert PEM to PFX and import it but mongod cannot find the certificate because of

--sslCertificateSelector subject=US

Mongod picks up the first certificate available with "US" and that one doesn't have an associated private key. I wonder why that happens? 

[jeremyhagand61]

Pure speculation, since I didn't take the time to test, but I would say & (command concatenation) or ^ (MS-DOS escape character)

[jeremyhagand61 · Accepted solution]

See my answer in the post