Failed to start KV store

[corti77]

Hi,

I run splunk 9.0.8 and after an issue with our storage (LUN full). I had to full scan the disk and successfully repaired the filesystem.

splunk starts now but there is a persistent error with the KV store. the status is the following:

show kvstore-status

This member:
backupRestoreStatus : Ready
disabled : 0
guid : E59FAAA8-D66A-498E-8DDF-0F5C29866F95
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger

Any suggestion on how could get an operational status?

many thanks.

Jose

[bvasir]

We are doing a dry run of a spunk 9.0.2 upgrade to 9.2.4 
and when running the splunk show kvstore-status just get status starting

How do we get this started?
Note in mind that we will be running this in prod in the near future

/opt/splunk/bin/splunk show kvstore-status
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

This member:
backupRestoreStatus : Ready
disabled : 0
guid : 9AEF8531-6F71-46C8-AC9F-F4EEE7FFE8DB
port : 7511
standalone : 0
status : starting
storageEngine : wiredTiger

[bvasir]

/opt/splunk/bin/splunk show kvstore-status
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Your session is invalid.  Please login.
Splunk username: admin
Password:

 This member:
                           backupRestoreStatus : Ready
                                      disabled : 0
                                          guid : 9AEF8531-6F71-46C8-AC9F-F4EEE7FFE8DB
                                          port : 7511
                                    standalone : 0
                                        status : starting
                                 storageEngine : wiredTiger

[corti77]

just voted, thanks a lot

[gcusello · Accepted solution]

Hi @corti77 ,

you have to reset the mongod.lock:

• stop splunkd
• delete $SPLUNK_HOME\var\lib\splunk\kvstore\mongo\mongod.lock"
• start splunkd

I asked to Splunk ideas to create a script od a dedicated solution to this frequent issue, if you think as me that's important, please vote for this idea at https://ideas.splunk.com/ideas/EID-I-2058

Ciao.

Giuseppe

[ringoshimaki]

how would I do this process in Windows Server 2019 running Splunk?

[Ogorek]

Hello, what to do when my kvstore folder just vanished? It do not create again after restart. Tried to create new folder with splunk privileges but it wont help. Do u have any idea how to reapir this?