Kvstore failed

[egko]

On my current machine, Kvstore is failing.
When I restart Splunk, the Kvstore status is "Ready." However, when I click the Audit Log tab in ES, the status changes to "Failed." This makes it impossible to access other Kvstore-related functions, such as incident review in ES.

I tried changing the server.pem file to a different extension and restarting, as well as changing the mongod.lock and splunk.key files to different extensions and restarting. I also tried changing all configuration files, but nothing worked.

I'm wondering if there are any other solutions.

Please help.

Splunk version : 8.1.10.1
Splunk Enterprise Security: 7.0.1

ERROR-Log
Failed to execute KVstore lookups External command based lookup 'goverence_lookup' is not available because KVstore initialization has failed, Contact your system admin...
Failed to create kvstore lookup

[PrewinThomas]

@egko 

Your Splunk Enterprise 8.1x is outdated, and Splunk Enterprise Security 7.0.1 is likely incompatible, causing the KV Store to fail (from "Ready" to "Failed") when ES loads.

Upgrade Splunk Enterprise, then upgrade ES to a compatible version.

#https://help.splunk.com/en/splunk-enterprise/get-started/install-and-upgrade/9.0/upgrade-or-migrate-...

Regards,
Prewin
🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[thahir]

@egko  Take the backup of KV store
ref: https://help.splunk.com/en/data-management/splunk-enterprise-admin-manual/9.4/administer-the-app-key... 

and try to clean the kvstore and restart the splunk.

check the status 

splunk show kvstore-status --verbose

splunk stop
splunk clean kvstore --local
splunk start

please refer the below document as well for more details about kv store troubleshooting

https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.0/welcome-to-splunk-enterpris... 

[egko]

The mongod.lock file is 0 bytes no matter how many times I restart it.

[thahir]

@egko  remove the lock file and clean up the kvstore and do the restart