KVstore failed to upgrade to 7.0 from 4.2 in Splunk enterprose version 9.4.1

[debsili05]

This is what we are getting 
KVStore Upgrade Status:
Upgrade Status: kvstore_upgrade_aborted
Start Time: Mon Oct 6 21:52:15 2025
End Time: Mon Oct 6 21:52:31 2025
Upgrade ID: 1759801935
Peer Retries: 1
Cluster Percentage: 100
Upgrade Version: 7.0

We even enable debug mode. We tried everything what we could as suggested by Splunk team. Nothing worked. Is anyone having the same issue as us?
Latest splunk support asked us to go from 4.2 to 4.4 and 5 and so on. But we really doubt if that would help. 
Is there anyone in similar situation as us?
Btw, its only happening in our PROD SH. Test and Dev went good.

[marnall]

Is your prod environment using a different SSL certificate than your test and dev environments? I have had headaches when using a cert that worked fine on version 9.3.x, but suddenly the KVStore failed on version 9.4.x because it required the SSL certificate to have clientAuth capability to start the Kvstore. It may be worth checking that you are not using the same cert config in server.conf and web.conf.

To solve this I used the default Splunk SSL cert in server.conf stanzas, but a custom cert in web.conf.

If this is the problem on your prod server, you might be able to find internal logs indicating the use of your web cert to communicate with the kvstore by searching:

index=_internal <yourcertfilename>

[PrewinThomas]

@debsili05 

Check your mongod.log and kvstore errors in the log. And what its showing when you check kvstore status

./splunk show kvstore-status

Also run below and see what its showing

./splunk start-standalone-upgrade kvstore -version 7.0 -dryRun true

Splunk recommended safest path to follow, which you can try. Also check for corrupted or oversized collections in PROD that may not exist in TEST/DEV.

Regards,
Prewin
If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[PickleRick]

Ok, so you enabled the debug mode and what did it tell you? Just turning it on doesn't fix anytning, it just makes it log debug information (hence the name). Your whole process aborted in 16 seconds. That's kinda quick.

You're saying your dev and test environments upgraded ok. Are they identically configured? If so then probably there's somehting with the contents of your kvstore database. I'd try to backup it, restore on a clean 4.2 env and try upgrading it.