Splunk Search Head Cluster Upgrade

[himanshu2]

I have a 2 search heads in a Splunk SH cluster in the dev environment. Recently, I upgraded Splunk from 9.3.8 to 9.4.11 version but kvstore didn't came up which I believe most probably due to Splunk restart in between /both Splunk SHs were stopped at the same time during upgrade activity.

TO resolve kvstore issues, I took backup of mongo from /opt/splunk/var/lib/splunk/kvstore/ and then restarted Splunk in both the servers as well, but nothing helped.

I disabled kvstore because it was not getting upgraded to 7.0.28 version. Now, when I checked SH Cluster status, for one of the SH, it is stuck at last_conf_replication : Pending.

I understand, we should have at least 3 SHs in a cluster but this is for testing purposes only and it was working fine with 2 SHs in the past. 
How can I resolve the SH Cluster and kvstore issues in this case?
@PickleRick @gcusello 

The mongod.log contents in the SH, where I don't see pending status 

[himanshu2]

Sorry, but this didn't work.

I tried everything but still kvstore is not upgraded. It has probably something to do with just two search heads in the cluster, but I am not really sure.

[gcusello]

Hi @himanshu2 ,

please don't call people in questions!

Anyway, following the documentation, it's a known issue, you must follow the documentation during upgrade from 9.3.x to 9.4.x:

https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/administer-the-app-key-valu...

Ciao.

Giuseppe