Failing KVstore after upgrade

[Dov1]

Hello,

I have splunk installed on 3 servers (searchhead, index, HF) on windows server.

I upgrade from 8.2.x to 9.2.1 - on the search head and index everything is working - including the kvstore (it was upgraded to wiredTiger before the upgrade.

BUT - on the HF the kvstore failing.

In the mongoDB log file I can see:

CONTROL [main] Failed global initialization: InvalidSSLConfiguration: Could not read private key attached to the selected certificate, ensure it exists and check the private key permissions

splunk show kvstore-status --verbose show:

This member:
backupRestoreStatus : Ready
disabled : 0
featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]
guid : xxxxxxxxxxxxxxxxxxxx
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger

I tried to:

1. Delete the server.pem file and also splunk clean kvstore --local but still the same error.
2. Commenting out the "sslPassword" under the stanza "[sslConfig]"  in the server.conf  didn't help.
3. The pfx file was added in the Windows certificate store - but not sure the right why.

I will be happy for any help.

[Hiattech]

I am also experiencing this issue, though it occurred after moving from Ubuntu to a stigged RHEL 9.  I am curious on what the fix was for you because I've tried everything I can think of.

[davidoff96]

Which server.pem file did you delete? You should also run a btool to see what cert is being used:

$SPLUNK_HOME/bin/splunk btool server list --debug | grep -i "ssl"

[shivanshu1593]

Is this issue resolved now or do you need more help? This is the issue with the key of the certificate of KVstore.