KV Store Upgrade Failure from 4.17 to 7.x During Splunk Enterprise Indexer Upgrade from 9.3.2 to 9.4.x

[Narendra_Rao]

Environment:

• Product: Splunk Enterprise (Indexer)
• Deployment: On-premises
• Current Version: 9.3.2
• Target Version: 9.4.x (tested 9.4.0, 9.4.2)
• Current KV Store Version: MongoDB 4.17
• Expected KV Store Version: MongoDB 7.x (per documentation)

Issue Summary: Experiencing KV Store upgrade failures when upgrading Splunk Enterprise Indexer from 9.3.2 to any 9.4.x version. According to Splunk documentation, the upgrade from 9.3.x to 9.4.x should be seamless with automatic KV Store upgrade from MongoDB 4.x to 7.x. Both automatic and manual KV Store upgrade approaches have failed.

Errors sample:
- 

alled Result::unwrap() on an Err value: UpgradeError { details: "Error updating status to 'INITIAL_UPGRADE_SEQUENCE' on 127.0.0.1:8191 document: Error { kind: Write(WriteError(WriteError { code: 11000, code_name: None, message: "E11000 duplicate key error collection: migration_metadata.migration_metadata index: id dup key: { _id: \"127.0.0.1:8191\" }", details: None })), labels: {}, wire_version: None, source: None }", kind: LocalError }

- Failed to upgrade KV Store to the latest version. KV Store is running an old version, 4.2. Resolve upgrade errors and try to upgrade KV Store to the latest version again.

Any others wiredTiger etc all might be wild fire and relevant Tried manuallay and also ansible automation both( same steps )

Question:

1. Why is KV Store upgrading to 4.25 instead of directly to 7.x as documented?
2. How to come out as we have big infra and we need to upgrade etc ?

[Subbhaiya]

....

[SanjayReddy]

Hi   @Narendra_Rao 

for upgrading to KV store 7.x

first kV store need to be in 4.2x, by looking at your kv store version its in 4.1.7 

this is reason upgrade is failing 

you need to upgrade to 4.2x. then go for 7.x 
Upgarde document to 4.2.x
https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.3/administer-the-app-key-valu...

then upgrade document to 7.x

https://docs.splunk.com/Documentation/Splunk/9.4.2/Admin/MigrateKVstore#Prepare_for_the_upgrade