For anyone upgrading to 9.4.1 and getting this, please read. I had this exact problem when testing (upgrading from 9.3.1 ro 9.4.1) in a sandbox environment. After many "failed" attempts I realized this is actually just a normal status and part of the kvstore upgrade process. What actually happens is that the kvstore upgrades in steps. From 4.x -> 5.x -> 6.x -> 7.x During these step, the status of the kvstore is in failed state. featureCompatibilityVersion : 5.0
...
status : failed
...
serverVersion : 5.0.26 featureCompatibilityVersion : 6.0
...
status : failed
...
serverVersion : 6.0.15 featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]
...
status : failed You also cannot rely on the "splunk show standalone-kvupgrade-status" command: /opt/splunk/bin/splunk show standalone-kvupgrade-status
Unable to read mongo database version. Check KV Store health. Just ignore all of these and allow the sytem to run for some +5 minutes. (do not stop splunk!) And then finally the upgrade completes and the status goes to ready. featureCompatibilityVersion : 7.0
...
status : ready
...
serverVersion : 7.0.14
... View more