topic Re: KV store termination issue frequently in Splunk Enterprise

KV store termination issue frequently

uagraw01 — Mon, 10 Feb 2025 16:31:35 GMT

Dear Splunkers!!

Following the migration of our Splunk server from version 8.1.1 to 9.1.1, we have encountered persistent KV Store failures. The service terminates unexpectedly multiple times post-migration.

Issue Summary:

As a workaround, I renewed the server.pem certificate and rebuilt the MongoDB folder.
This temporarily resolves the issue, and KV Store starts working as expected.
However, the corruption reoccurs the following day, requiring the same manual interventions.

Request for Permanent Resolution:

I seek a permanent fix to prevent KV Store from repeatedly failing. Kindly provide insights into the root cause and recommend a robust solution to ensure KV Store stability post-migration.

Looking forward to your expert guidance.

Re: KV store termination issue frequently

livehybrid — Mon, 10 Feb 2025 16:44:11 GMT

Hi @uagraw01

Are there any mongo/kvstore logs in $SPLUNK_HOME/var/log/splunk/splunkd.log or mongod.log with any error/critical/fatal or maybe even warning messages?

What process did you take to rebuild the mongoDB folder?

Thanks, hopefully we can help get to the bottom of it!

Re: KV store termination issue frequently

uagraw01 — Mon, 10 Feb 2025 17:36:40 GMT

I just removed complete kvstore folder from "/opt/splunk/var/lib/splunk/" after taking the backup and restart the splunk services.

Re: KV store termination issue frequently

uagraw01 — Tue, 18 Feb 2025 13:30:59 GMT

@livehybrid For your information. I have changed the kvstore port from 8191 to 8192 and its start working properly since then.

Re: KV store termination issue frequently

anhduc2901 — Thu, 26 Jun 2025 03:38:56 GMT

@uagraw01 how you change the port ? i mean when i reconf in /splunk/etc/system/local/server.conf with port = 8192 under stanza [kvstore] then the kvstore cant be enable anymore (status = failed)