KVSTORE FAILED TO START ISSUE

[dy1]

KV Store changed status to failed. KVStore process terminated.. 10/2/2025, 12:23:23 am

Failed to start KV Store process. See mongod.log and splunkd.log for details. 10/2/2025, 12:23:23 am

KV Store process terminated abnormally (exit code 4, status PID 6147 killed by signal 4: Illegal instruction). See mongod.log and splunkd.log for details.

 

 

this above mention issues are showing .

 

#kvstore @kvstore @splunk

[myitlab42000]

hi,

could you check this requirements. For example, is your cpu supported avx / avx2 instructions, if yes, is it enabled ?

https://docs.splunk.com/Documentation/Splunk/9.4.0/Admin/MigrateKVstore

https://www.mongodb.com/docs/manual/administration/production-notes/

i hope this help

[kiran_panchavat]

@dy1 

See the status of the KV store by using the following command.

/opt/splunk/bin/splunk show kvstore-status -auth <user_name>:<password>

Review the mongod.log and splunkd.log files for more detailed error messages.

If there's a lock file causing the issue, you can remove it:

sudo rm -rf /xxx/kvstore/mongo/mongod.lock

Renaming the current MongoDB folder can help reset the KV Store.

mv $SPLUNK_HOME/var/lib/splunk/kvstore/mongo $SPLUNK_HOME/var/lib/splunk/kvstore/mongo.old

Steps:-

Stop Splunk
Rename the current mongo folder to old
Start Splunk
And you will see a new Mongo folder created with all the components.

[richgalloway]

What does mongod.log say?

What version of Splunk?  What version of KVStore?  Are you using mmapv1 or wiredTiger?

[Dikshi]

using wiredtiger

[richgalloway]

Thanks.  Please answer the other three questions.