Hi @Mfmahdi Please do not tag/call out specific users on here - there are lots of people monitoring for questions being raised and those you have tagged do have day jobs and other priorities so you...
See more...
Hi @Mfmahdi Please do not tag/call out specific users on here - there are lots of people monitoring for questions being raised and those you have tagged do have day jobs and other priorities so you risk your question being missed. To troubleshoot the KV Store initialization issue, start by examining the logs on the search head cluster members for specific errors. | rest /services/kvstore/status | fields splunk_server, current* Then check on each SHC member: ps -ef | grep mongod
# Check mongod logs for errors
tail -n 200 $SPLUNK_HOME/var/log/splunk/mongod.log
# Check splunkd logs for KV Store related errors
grep KVStore $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -n 200 Verify mongod Process: Ensure the mongod process, which underlies the KV Store, is running on the search head members. Use the ps command or your operating system's equivalent. If it's not running, investigate why using the logs. Check Cluster Health: Ensure the search head cluster itself is healthy using the Monitoring Console or the CLI command splunk show shcluster-status run from the captain. KV Store issues can sometimes be symptomatic of underlying cluster communication problems. From your screenshot it looks like this is showing as starting state, so hopefully the logs shine some light on the issue. Check Resources: Verify sufficient disk space, memory, and CPU resources on the search head cluster members, particularly on the node currently acting as the KV Store primary. Focus on the error messages found in mongod.log and splunkd.log as they usually pinpoint the root cause (e.g., permissions, disk space, configuration errors, corrupted files). If the logs indicate corruption or persistent startup failures that restarts don't resolve, you may need to consider more advanced recovery steps, potentially involving Splunk Support. USeful docs which might help: Splunk Docs: Troubleshoot the KV Store Splunk Docs: About the KV Store Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing