KV Store lookup failing with error about KV store initialization failure

[nnmiller]

KV store lookups are failing with the following error:

Error in 'inputlookup' command: External command based lookup 'kvstore_lookup' is not available because KV Store initialization has failed. 

Verified no orphaned mongod.lock file or firewall port issues per Splunk Answer 208349. Attempted manual restart of mongod with

./mongod --fork --logpath $SPLUNK_HOME/var/log/splunk/test_mongodb.log
about to fork child process, waiting until server is ready for connections.
forked process: 4234
ERROR: child process failed, exited with error number 100

Searching for above error, noted a post on StackOverflow where permissions were a problem, so I verified permissions. Mongod still did not start.

[nnmiller · Accepted solution]

Closer review of mongod.log showed the following errors:

mongod.log: 2016-04-27T16:42:40.111Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
mongod.log: 2016-04-27T16:42:40.129Z I CONTROL  dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

First line appears to be spurious, second log entry was the problem. Splunk was looking at /opt/splunk/etc/auth/server.pem, showing in successful, earlier starts of mongod. $SPLUNK_HOME/etc/auth/server.pem is the default path and cert. Verified expiry by running: $SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

Replaced with a newly generated cert, kv store started per norm.

In recent versions of Splunk you can regenerate the server's certificate using the following steps:

• Run the following command to update the server certificate:

  /opt/splunk/bin/splunk createssl server-cert -d <path_to_rootCA> -n server.pem -c <dns_name_of_host> -p
  

  The default location for the rootCA files is $SPLUNK_HOME/etc/auth/.

• If you know the current sslKeysfilePassword for the search head, you can use the same password when creating the new certificate and won't need to change anything else. If you do not know the search head's plaintext sslKeysfilePassword then you will need to edit $SPLUNK_HOME/etc/system/local/server.conf and update the sslKeysfilePassword in the [sslConfig] stanza to match the new passphrase you used when generating the new server cert.

• In older versions of Splunk, you can use the script $SPLUNK_HOME/bin/genSignedServerCert.sh

[briancronrath]

You are a life saver

[smoir_splunk]

In newer versions of Splunk, sslKeysfilePassword is deprecated in favor of sslPassword. See https://docs.splunk.com/Documentation/Splunk/7.2.4/Admin/Serverconf for more info about these settings.

[varunCarbyne]

We faced the SSL problem and when we try to check the certificate validity, we see

notAfter= some date of 2020 GMT

Please suggest any other option for same

[Shephali]

Hi @varunCarbyne ,

Did you figure out the solution?
I have a similar issue at my end.SSL Certificate validity is fine,still KV initialization failing

[Waltersr24]

where would you go to generate the new cert? im new with Splunk

[ykpramodh]

Hi,

We faced the kv store problem where kv store stays in "starting" status. Checked the mongod.log and identified the SSL problem suggested above.

We faced the SSL problem and when we try to check the certificate validity, we see

notAfter=Dec 9 19:01:45 2019 GMT

Can you please suggest any other options we should try?

[nnmiller]

If KV Store stays in "Starting" status, then you have a different problem, assuming mongod is actually running. ERROR: child process failed, exited with error number 100 is a generic error.

I would suggest opening a support ticket if you haven't resolved the problem already.