Kvstore Status:failed

[Spinner79]

Hi all,

Please help. 

I have a AIO Splunk Enterprise Ver: 8.1.2 setup running on windows server, i have done the following upgrade:

1. Upgrade from Ver: 8.1.2 to ver: 8.2.5.

2. checked kvstore already migrated to wireTiger. So no migration done on my end.

3. Upgrade from Ver: 8.2.5 to Ver: 9.0.5.

After start up checked kvstore with the command "splunk show kvstore-status --verbose", as shown below is the result:

WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliverifyServerName for details.

This member:

                                    BackupRestoreStatus : Ready

                                                               disables : 0

                     featureCompaitibilityVersion : An error occured during the last operation ('getParameter', domain:  '15', code: '13053'): No Suitable servers found: 'serxxxx SelectionTimeoutMS' expired: [connection closed calling ismaster on '127.0.0.1:8191']

                                                                    guid : DAxxxxxx-xxxx-xxx-xxxx-xxxxxxxxxxxx

                                                                    port : 8191

                                                       standalone : 1

                                                                 status : failed

                                                 storageEngine : wireTiger

Can anyone advise why after upgraded to Ver: 9.0.5 kvstore failed? i have tried the following but the status still showed failed:

Point 2: Take KV store backup

2.1. stop Splunk.
2.2. take backup of the DB (keep it safe - may be needed in emergency):
2.3. cp -r $SPLUNK_DB/kvstore/mongo /backup
2.4. Rename mongod.lock (Location: $SPLUNK_DB/kvstore/mongod) : mongod.lock_bkp

Point 3: clean up KVStore Splunk

3.1 Stop the member: ./splunk stop
3.2 Clean the member's kvstore: ./splunk clean kvstore --local
3.3 start Splunk

Point 4. Check kvstore status on each node.
./splunkshow kvstore-status

Please help. Thank you

[cmeisch]

Just happened to us now... Do we know if this fixed it and\or what was the initial cause.  This was just after a splunkd restart.

[Spinner79]

Hi,

I tried the follow the link https://splunkonbigdata.com/failed-to-start-kv-store-process-see-mongod-log-and-splunkd-log-for-deta... , surprisingly after restart everything back to normal.

[dwthomas16]

Feels like I tried every suggestion here, from renaming the cert, generating my own, messing around with the windows cert store and the .conf files... And this solution finally worked! Thank you so much!!

[Spinner79]

Glad that you have managed to resolve the issue like me.

I went through high & low searching for the solutions as well.

Luckily managed to Google out the link I sent you and resolved the issue.

@dwthomas16 Happy Splunking.

[dwthomas16]

Were you able to find a solution to this? I'm having the same issue.