KV Store Errors : KV Store changed status to failed. Failed to start KV Store process. Getting KV store errors without using KV store

[Ankitha_d]

I have a standalone Search head catering to Indexer cluster with 2 indexers.
On both SH and IDX, we get KV store initialization failure.And suggestion is to check mongod.log and splunkd.log for errors
But mongod.log and splunkd.log has no specific errors.

And we are not even using kvstore , but still have errors on both SH and Indexers

mongod.log Errors

2018-05-02T11:24:17.028Z I JOURNAL [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal

2018-05-02T11:24:16.981Z I CONTROL [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "/opt/splunk/etc/auth/server.pem", PEMKeyPassword: "", allowInvalidHostnames: true,disabledProtocols: "noTLS1_0,noTLS1_1", mode: "preferSSL", sslCipherConfig: "xxx" }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "30B62029-B878-4421-B99F-686EA7CC8A8A" }, security: { javascriptEnabled: false, keyFile: "/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "/opt/splunk/var/lib/splunk/kvstore/mongo", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }

2018-05-02T11:24:16.981Z I CONTROL [initandlisten] MongoDB starting : pid=20725 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=xxxxx

splunkd.log Errors

05-02-2018 11:29:17.341 +0000 ERROR KVStoreBulletinBoardManager - Failed to start KV Store process. See mongod.log and splunkd.log for details.
05-02-2018 11:29:17.341 +0000 ERROR KVStoreBulletinBoardManager - KV Store changed status to failed. Failed to start KV Store process. See mongod.log and splunkd.log for details.
05-02-2018 11:29:17.341 +0000 ERROR KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.
05-02-2018 11:29:17.341 +0000 ERROR KVStoreConfigurationProvider - Could not get pint from mongod.
05-02-2018 11:24:16.095 +0000 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)

05-03-2018 03:06:06.660 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py" splunklib.binding.HTTPError: HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator.

Things I have already tried :

1>Have given 600 permission to splunk.key and restarted. All files here have read and write perm,even mongod.lock
2>The certificates are all valid in /etc/auth .Have also deleted server.pem and restarted to generate new server.pem.
3>Stopped the SH and ran splunk clean kvstore --local and restarted , only to find the same error again

Can anyone please help me out with this issue????

[Michael · Accepted solution]

I solved this by generating new key with:

/opt/splunk/bin/splunk createssl server-cert 3072 -d /opt/splunk/etc/auth/ -n server -c {your domain name}

/opt/splunk/bin/splunk restart

[charlesh]

I have ran through everything the same including generating the new key. Still the KV Store will not start and is
failed state.

"warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter"

Do I need go through and redo all the certs because they are not signed?

Any other things worth checking?

Help is much appreciated!!!

[alexadao]

Thank you, That works for me.
/opt/splunk/bin/splunk createssl server-cert 3072 -d /opt/splunk/etc/auth/ -n server -c {your domain name}