KVStore is not ready. Token auth system will not work.

[whrg]

When I navigate to Settings > Tokens, I get this error message:

KVStore is not ready. Token auth system will not work.

Splunk logs shows this:

ERROR JsonWebToken [233289 TcpChannelThread] - KVStore is not ready. Token auth system will not work.
ERROR KVStoreConfigurationProvider [233052 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=KVStore service will not start because kvstore process terminated
ERROR KVStoreBulletinBoardManager [233053 MongodLogThread] - KV Store changed status to failed. KVStore process terminated..

How can this be fixed?

[whrg · Accepted solution]

I found the solution which I came across here: https://community.splunk.com/t5/Security/How-do-I-renew-an-expired-Splunk-Certificate/m-p/389701

Turns out, the Splunk certificate was expired. This is how I checked:

$ openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem
notAfter=Feb 27 13:56:21 2024 GMT

To get a new certificate, I removed the old certificate and restarted Splunk (a new certificate will be created when Splunk starts):

$ mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem.backup

Now Settings > Tokens is working again.

[isoutamo]

Hi
Have you looked from mongod.log (or something similar) why mongod didn’t start?
r. Ismo