KV store termination issue frequently

[uagraw01]

Dear Splunkers!!

Following the migration of our Splunk server from version 8.1.1 to 9.1.1, we have encountered persistent KV Store failures. The service terminates unexpectedly multiple times post-migration.

Issue Summary:

• As a workaround, I renewed the server.pem certificate and rebuilt the MongoDB folder.
• This temporarily resolves the issue, and KV Store starts working as expected.
• However, the corruption reoccurs the following day, requiring the same manual interventions.

Request for Permanent Resolution:

I seek a permanent fix to prevent KV Store from repeatedly failing. Kindly provide insights into the root cause and recommend a robust solution to ensure KV Store stability post-migration.

Looking forward to your expert guidance.

[uagraw01]

@livehybrid  For your information. I have changed the kvstore port from 8191 to 8192 and its start working properly since then.

[anhduc2901]

@uagraw01 how you change the port ? i mean when i reconf in /splunk/etc/system/local/server.conf with port = 8192 under stanza [kvstore] then the kvstore cant be enable anymore (status = failed) 

[livehybrid]

Hi @uagraw01 

Are there any mongo/kvstore logs in $SPLUNK_HOME/var/log/splunk/splunkd.log or mongod.log with any error/critical/fatal or maybe even warning messages?

What process did you take to rebuild the mongoDB folder?

Thanks, hopefully we can help get to the bottom of it!

[uagraw01]

I just removed complete kvstore folder from "/opt/splunk/var/lib/splunk/" after taking the backup and restart the splunk services.