Knowledge Management

KV Store Errors : KV Store changed status to failed. Failed to start KV Store process. Getting KV store errors without using KV store

Ankitha_d
Path Finder

I have a standalone Search head catering to Indexer cluster with 2 indexers.
On both SH and IDX, we get KV store initialization failure.And suggestion is to check mongod.log and splunkd.log for errors
But mongod.log and splunkd.log has no specific errors.

And we are not even using kvstore , but still have errors on both SH and Indexers

mongod.log Errors

2018-05-02T11:24:17.028Z I JOURNAL [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal

2018-05-02T11:24:16.981Z I CONTROL [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "/opt/splunk/etc/auth/server.pem", PEMKeyPassword: "", allowInvalidHostnames: true,disabledProtocols: "noTLS1_0,noTLS1_1", mode: "preferSSL", sslCipherConfig: "xxx" }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "30B62029-B878-4421-B99F-686EA7CC8A8A" }, security: { javascriptEnabled: false, keyFile: "/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "/opt/splunk/var/lib/splunk/kvstore/mongo", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }

2018-05-02T11:24:16.981Z I CONTROL [initandlisten] MongoDB starting : pid=20725 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=xxxxx

splunkd.log Errors

05-02-2018 11:29:17.341 +0000 ERROR KVStoreBulletinBoardManager - Failed to start KV Store process. See mongod.log and splunkd.log for details.
05-02-2018 11:29:17.341 +0000 ERROR KVStoreBulletinBoardManager - KV Store changed status to failed. Failed to start KV Store process. See mongod.log and splunkd.log for details.
05-02-2018 11:29:17.341 +0000 ERROR KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.
05-02-2018 11:29:17.341 +0000 ERROR KVStoreConfigurationProvider - Could not get pint from mongod.
05-02-2018 11:24:16.095 +0000 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)

05-03-2018 03:06:06.660 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py" splunklib.binding.HTTPError: HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator.

Things I have already tried :

1>Have given 600 permission to splunk.key and restarted. All files here have read and write perm,even mongod.lock
2>The certificates are all valid in /etc/auth .Have also deleted server.pem and restarted to generate new server.pem.
3>Stopped the SH and ran splunk clean kvstore --local and restarted , only to find the same error again

Can anyone please help me out with this issue????

1 Solution

Michael
Contributor

I solved this by generating new key with:

/opt/splunk/bin/splunk createssl server-cert 3072 -d /opt/splunk/etc/auth/ -n server -c {your domain name}

/opt/splunk/bin/splunk restart

View solution in original post

Michael
Contributor

I solved this by generating new key with:

/opt/splunk/bin/splunk createssl server-cert 3072 -d /opt/splunk/etc/auth/ -n server -c {your domain name}

/opt/splunk/bin/splunk restart

charlesh
Loves-to-Learn

I have ran through everything the same including generating the new key. Still the KV Store will not start and is
failed state.

"warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter"

Do I need go through and redo all the certs because they are not signed?

Any other things worth checking?

Help is much appreciated!!!

0 Karma

alexadao
Observer

Thank you, That works for me.
/opt/splunk/bin/splunk createssl server-cert 3072 -d /opt/splunk/etc/auth/ -n server -c {your domain name}

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...