topic Re: Kvstore Status:failed in Knowledge Management

Kvstore Status:failed

Spinner79 — Fri, 07 Jul 2023 02:26:09 GMT

Hi all,

Please help.

I have a AIO Splunk Enterprise Ver: 8.1.2 setup running on windows server, i have done the following upgrade:

1. Upgrade from Ver: 8.1.2 to ver: 8.2.5.

2. checked kvstore already migrated to wireTiger. So no migration done on my end.

3. Upgrade from Ver: 8.2.5 to Ver: 9.0.5.

After start up checked kvstore with the command "splunk show kvstore-status --verbose", as shown below is the result:

WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliverifyServerName for details.

This member:

BackupRestoreStatus : Ready

disables : 0

featureCompaitibilityVersion : An error occured during the last operation ('getParameter', domain: '15', code: '13053'): No Suitable servers found: 'serxxxx SelectionTimeoutMS' expired: [connection closed calling ismaster on '127.0.0.1:8191']

guid : DAxxxxxx-xxxx-xxx-xxxx-xxxxxxxxxxxx

port : 8191

standalone : 1

status : failed

storageEngine : wireTiger

Can anyone advise why after upgraded to Ver: 9.0.5 kvstore failed? i have tried the following but the status still showed failed:

Point 2: Take KV store backup

2.1. stop Splunk.
2.2. take backup of the DB (keep it safe - may be needed in emergency):
2.3. cp -r $SPLUNK_DB/kvstore/mongo /backup
2.4. Rename mongod.lock (Location: $SPLUNK_DB/kvstore/mongod) : mongod.lock_bkp

Point 3: clean up KVStore Splunk

3.1 Stop the member: ./splunk stop
3.2 Clean the member's kvstore: ./splunk clean kvstore --local
3.3 start Splunk

Point 4. Check kvstore status on each node.
./splunkshow kvstore-status

Please help. Thank you

Re: Kvstore Status:failed

dwthomas16 — Mon, 28 Aug 2023 13:15:14 GMT

Were you able to find a solution to this? I'm having the same issue.

Re: Kvstore Status:failed

Spinner79 — Tue, 29 Aug 2023 02:00:58 GMT

Hi,

I tried the follow the link https://splunkonbigdata.com/failed-to-start-kv-store-process-see-mongod-log-and-splunkd-log-for-details/ , surprisingly after restart everything back to normal.

Re: Kvstore Status:failed

dwthomas16 — Wed, 30 Aug 2023 13:55:34 GMT

Feels like I tried every suggestion here, from renaming the cert, generating my own, messing around with the windows cert store and the .conf files... And this solution finally worked! Thank you so much!!

Re: Kvstore Status:failed

Spinner79 — Thu, 31 Aug 2023 03:54:07 GMT

Glad that you have managed to resolve the issue like me.

I went through high & low searching for the solutions as well.

Luckily managed to Google out the link I sent you and resolved the issue.

@dwthomas16 Happy Splunking.

Re: Kvstore Status:failed

cmeisch — Mon, 12 Aug 2024 17:07:49 GMT

Just happened to us now... Do we know if this fixed it and\or what was the initial cause. This was just after a splunkd restart.