topic Re: KVStore is not ready. Token auth system will not work. in Splunk Enterprise

KVStore is not ready. Token auth system will not work.

whrg — Thu, 29 Feb 2024 12:26:31 GMT

When I navigate to Settings > Tokens, I get this error message:

KVStore is not ready. Token auth system will not work.

Splunk logs shows this:

ERROR JsonWebToken [233289 TcpChannelThread] - KVStore is not ready. Token auth system will not work. ERROR KVStoreConfigurationProvider [233052 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=KVStore service will not start because kvstore process terminated ERROR KVStoreBulletinBoardManager [233053 MongodLogThread] - KV Store changed status to failed. KVStore process terminated..

How can this be fixed?

Re: KVStore is not ready. Token auth system will not work.

isoutamo — Thu, 29 Feb 2024 14:20:15 GMT

Hi
Have you looked from mongod.log (or something similar) why mongod didn’t start?
r. Ismo

Re: KVStore is not ready. Token auth system will not work.

whrg — Tue, 05 Mar 2024 12:48:25 GMT

I found the solution which I came across here: https://community.splunk.com/t5/Security/How-do-I-renew-an-expired-Splunk-Certificate/m-p/389701

Turns out, the Splunk certificate was expired. This is how I checked:

$ openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem notAfter=Feb 27 13:56:21 2024 GMT

To get a new certificate, I removed the old certificate and restarted Splunk (a new certificate will be created when Splunk starts):

$ mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem.backup

Now Settings > Tokens is working again.