Installation

Failing KVstore after upgrade

Dov1
Observer

Hello,

I have splunk installed on 3 servers (searchhead, index, HF) on windows server.

I upgrade from 8.2.x to 9.2.1 - on the search head and index everything is working - including the kvstore (it was upgraded to wiredTiger before the upgrade.

BUT - on the HF the kvstore failing.

In the mongoDB log file I can see:

CONTROL [main] Failed global initialization: InvalidSSLConfiguration: Could not read private key attached to the selected certificate, ensure it exists and check the private key permissions

splunk show kvstore-status --verbose show:

This member:
backupRestoreStatus : Ready
disabled : 0
featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]
guid : xxxxxxxxxxxxxxxxxxxx
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger

I tried to:

  1. Delete the server.pem file and also splunk clean kvstore --local but still the same error.
  2. Commenting out the "sslPassword" under the stanza "[sslConfig]"  in the server.conf  didn't help.
  3. The pfx file was added in the Windows certificate store - but not sure the right why.

I will be happy for any help.

Labels (2)
0 Karma

Hiattech
Explorer

I am also experiencing this issue, though it occurred after moving from Ubuntu to a stigged RHEL 9.  I am curious on what the fix was for you because I've tried everything I can think of.

0 Karma

davidoff96
Path Finder

Which server.pem file did you delete? You should also run a btool to see what cert is being used:

$SPLUNK_HOME/bin/splunk btool server list --debug | grep -i "ssl"

0 Karma

shivanshu1593
Builder

Is this issue resolved now or do you need more help? This is the issue with the key of the certificate of KVstore.

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...