topic Re: Splunk Search Head Cluster Upgrade in Deployment Architecture

Splunk Search Head Cluster Upgrade

himanshu2 — Mon, 18 May 2026 12:00:02 GMT

I have a 2 search heads in a Splunk SH cluster in the dev environment. Recently, I upgraded Splunk from 9.3.8 to 9.4.11 version but kvstore didn't came up which I believe most probably due to Splunk restart in between /both Splunk SHs were stopped at the same time during upgrade activity.

TO resolve kvstore issues, I took backup of mongo from /opt/splunk/var/lib/splunk/kvstore/ and then restarted Splunk in both the servers as well, but nothing helped.

I disabled kvstore because it was not getting upgraded to 7.0.28 version. Now, when I checked SH Cluster status, for one of the SH, it is stuck at last_conf_replication : Pending.

I understand, we should have at least 3 SHs in a cluster but this is for testing purposes only and it was working fine with 2 SHs in the past.
How can I resolve the SH Cluster and kvstore issues in this case?
@PickleRick @gcusello

The mongod.log contents in the SH, where I don't see pending status

Re: Splunk Search Head Cluster Upgrade

gcusello — Mon, 18 May 2026 10:41:20 GMT

Hi @himanshu2 ,

please don't call people in questions!

Anyway, following the documentation, it's a known issue, you must follow the documentation during upgrade from 9.3.x to 9.4.x:

https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/administer-the-app-key-value-store/upgrade-the-kv-store-server-version

Ciao.

Giuseppe

Re: Splunk Search Head Cluster Upgrade

himanshu2 — Tue, 19 May 2026 10:30:39 GMT

Sorry, but this didn't work.

I tried everything but still kvstore is not upgraded. It has probably something to do with just two search heads in the cluster, but I am not really sure.