topic Re: KVSTORE FAILED TO START ISSUE in Monitoring Splunk

KVSTORE FAILED TO START ISSUE

dy1 — Sun, 09 Feb 2025 19:07:49 GMT

KV Store changed status to failed. KVStore process terminated.. 10/2/2025, 12:23:23 am

Failed to start KV Store process. See mongod.log and splunkd.log for details. 10/2/2025, 12:23:23 am

KV Store process terminated abnormally (exit code 4, status PID 6147 killed by signal 4: Illegal instruction). See mongod.log and splunkd.log for details.

this above mention issues are showing .

#kvstore @kvstore @splunk

Re: KVSTORE FAILED TO START ISSUE

richgalloway — Sun, 09 Feb 2025 19:20:01 GMT

What does mongod.log say?

What version of Splunk? What version of KVStore? Are you using mmapv1 or wiredTiger?

Re: KVSTORE FAILED TO START ISSUE

kiran_panchavat — Mon, 10 Feb 2025 05:38:46 GMT

@dy1

See the status of the KV store by using the following command.

/opt/splunk/bin/splunk show kvstore-status -auth <user_name>:<password>

Review the mongod.log and splunkd.log files for more detailed error messages.

If there's a lock file causing the issue, you can remove it:

sudo rm -rf /xxx/kvstore/mongo/mongod.lock

Renaming the current MongoDB folder can help reset the KV Store.

mv $SPLUNK_HOME/var/lib/splunk/kvstore/mongo $SPLUNK_HOME/var/lib/splunk/kvstore/mongo.old

Steps:-

Stop Splunk
Rename the current mongo folder to old
Start Splunk
And you will see a new Mongo folder created with all the components.

Re: KVSTORE FAILED TO START ISSUE

Dikshi — Fri, 14 Feb 2025 20:32:09 GMT

using wiredtiger

Re: KVSTORE FAILED TO START ISSUE

richgalloway — Fri, 14 Feb 2025 20:59:18 GMT

Thanks. Please answer the other three questions.

Re: KVSTORE FAILED TO START ISSUE

myitlab42000 — Sat, 15 Feb 2025 12:40:14 GMT

hi,

could you check this requirements. For example, is your cpu supported avx / avx2 instructions, if yes, is it enabled ?

https://docs.splunk.com/Documentation/Splunk/9.4.0/Admin/MigrateKVstore

https://www.mongodb.com/docs/manual/administration/production-notes/

i hope this help