topic Failed to start KV store in Knowledge Management

Failed to start KV store

corti77 — Tue, 30 Apr 2024 23:33:33 GMT

Hi,

I run splunk 9.0.8 and after an issue with our storage (LUN full). I had to full scan the disk and successfully repaired the filesystem.

splunk starts now but there is a persistent error with the KV store. the status is the following:

show kvstore-status

This member:
backupRestoreStatus : Ready
disabled : 0
guid : E59FAAA8-D66A-498E-8DDF-0F5C29866F95
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger

Any suggestion on how could get an operational status?

many thanks.

Jose

Re: Failed to start KV store

gcusello — Wed, 01 May 2024 05:13:03 GMT

Hi @corti77 ,

you have to reset the mongod.lock:

stop splunkd
delete $SPLUNK_HOME\var\lib\splunk\kvstore\mongo\mongod.lock"
start splunkd

I asked to Splunk ideas to create a script od a dedicated solution to this frequent issue, if you think as me that's important, please vote for this idea at https://ideas.splunk.com/ideas/EID-I-2058

Ciao.

Giuseppe

Re: Failed to start KV store

corti77 — Mon, 06 May 2024 07:20:41 GMT

just voted, thanks a lot

Re: Failed to start KV store

Ogorek — Thu, 03 Oct 2024 18:49:35 GMT

Hello, what to do when my kvstore folder just vanished? It do not create again after restart. Tried to create new folder with splunk privileges but it wont help. Do u have any idea how to reapir this?

Re: Failed to start KV store

ringoshimaki — Thu, 08 May 2025 17:05:55 GMT

how would I do this process in Windows Server 2019 running Splunk?

Re: Failed to start KV store

bvasir — Mon, 12 May 2025 14:24:56 GMT

/opt/splunk/bin/splunk show kvstore-status WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. Your session is invalid. Please login. Splunk username: admin Password: This member: backupRestoreStatus : Ready disabled : 0 guid : 9AEF8531-6F71-46C8-AC9F-F4EEE7FFE8DB port : 7511 standalone : 0 status : starting storageEngine : wiredTiger

Re: Failed to start KV store

bvasir — Mon, 12 May 2025 14:27:55 GMT

We are doing a dry run of a spunk 9.0.2 upgrade to 9.2.4
and when running the splunk show kvstore-status just get status starting

How do we get this started?
Note in mind that we will be running this in prod in the near future

/opt/splunk/bin/splunk show kvstore-status
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

This member:
backupRestoreStatus : Ready
disabled : 0
guid : 9AEF8531-6F71-46C8-AC9F-F4EEE7FFE8DB
port : 7511
standalone : 0
status : starting
storageEngine : wiredTiger