Knowledge Management

Windows upgrade from 8.1.1 to 9.0: Why does it fail to start KV store process?

Sh4ne0
Explorer

I see lots of suggestions in the Community for Linux but not Windows. Has anyone resolved this on a production Windows 2016 Server?

Three errors when logging in to Splunk following the upgrade to 9.0

Failed to start the KV Store See mongod.log and splunkd.log for details.

KV Store changed status to failed. KV Store process terminated.

KV Store process terminated abnormally (exit code 1, status exited with code 1) See mongod.log and splunkd.log for details

Labels (1)
Tags (3)

NotSure
Explorer

I am having almost the exact same issue, the only minor difference is instead of:

"InvalidSSLConfiguration: Could not read private key attached to the selected certificate, ensure it exists and check the private key permissions"

I have an output of

"InvalidSSLConfiguration: Could not find private key attached to the selected certificate, ensure it exists and check the private key permissions"

I have verified the certificate and key are in the correct locations, have appropriate permissions and the certificate is not expired. This is on windows server 2019 standard. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Did you "See mongod.log and splunkd.log for details"?

kvstore failing at the start is often a case of expired certs.

0 Karma

Sh4ne0
Explorer

splunkd.log

07-05-2022 07:20:51.531 +1000 WARN ProcessTracker [10916 MainThread] - (child_15__Fsck) SSLOptions - server.conf/[sslConfig]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security

 

mongod.log

2022-07-04T06:03:51.553Z F CONTROL [main] Failed global initialization: InvalidSSLConfiguration: Could not read private key attached to the selected certificate, ensure it exists and check the private key permissions
2022-07-04T16:18:31.202+1000 W CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead.
2022-07-04T16:18:31.206+1000 W CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead.
2022-07-04T16:18:31.206+1000 W CONTROL [main] Option: sslAllowConnectionsWithoutCertificates is deprecated. Please use tlsAllowConnectionsWithoutCertificates instead.
2022-07-04T16:18:31.206+1000 W CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead.
2022-07-04T16:18:31.206+1000 W CONTROL [main] Option: sslAllowInvalidCertificates is deprecated. Please use tlsAllowInvalidCertificates instead.
2022-07-04T16:18:31.207+1000 W CONTROL [main] Option: sslCertificateSelector is deprecated. Please use tlsCertificateSelector instead.
2022-07-04T06:18:31.232Z W CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release.

0 Karma

lowcrawl
Explorer

Was there a fix for this?  I have the same warnings after upgrading from 8.2.9 to 9.0.4.1.   

0 Karma

simenhaugen
Explorer

@Sh4ne0 - Did you figure out what was causing these issues?

I'm seeing the same messages in mongod.log now after changing to new certs. 

0 Karma
Get Updates on the Splunk Community!

Join Us for Splunk University and Get Your Bootcamp Game On!

If you know, you know! Splunk University is the vibe this summer so register today for bootcamps galore ...

.conf24 | Learning Tracks for Security, Observability, Platform, and Developers!

.conf24 is taking place at The Venetian in Las Vegas from June 11 - 14. Continue reading to learn about the ...

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...