Unable to push shcluster bundles after upgrade- possible kvstore issue?

[dtow1]

I am unable to push shcluster bundles post an upgrade to 9.0.2 from 8.2.7. I have also completed the upgrade and migrated the KVstore without error and see the following expected settings:

serverVersion : 4.2.17

storageEngine : wiredTiger

The error I receive is:
"Error in pre-deploy check, uri=https://<HOST_NAME>/services/shcluster/captain/kvstore-upgrade/status, status=502, error=No error"

If I look in splunkd.log I get the following error for each attempt.
HttpClientRequest [2071959 TcpChannelThread] - Caught exception while parsing HTTP reply: Unexpected character while looking for value: '<'

The error from the actual command makes me think that there was an issue with the kvstore-upgrade that is just not showing.

[dtow1]

Additional details:

When I attempt to do the apply shcluster-bundle, if I look on the target search head in:

In  web_access.log I can see the get requests:
GET /en-US/services/shcluster/captain/kvstore-upgrade/status?output_mode=json HTTP/1.1" 404 18843

In web_service.log I can see the error that the path is not found.

error:321 - Masking the original 404 message: 'The path '/en-US/services/shcluster/captain/kvstore-upgrade/status' was not found.' with 'Page not found!' for security reason


Is there any way to disable this check in the apply shcluster-bundle command? Any way to determine why this page does not exist?

Thanks for any help.

[dtow1]

One last piece of information, I've found that if I modify the url to include the port, it is accessible:
https://<HOSTNAME>:8089/services/shcluster/captain/kvstore-upgrade/status

So it looks like this check is possibly failing due to an issue with the validation script.

I attempted this because every reference to this and similar checks in the migration documentation uses this port.

https://docs.splunk.com/Documentation/Splunk/9.0.2/Admin/MigrateKVstore

Is this a bug in 9.0.2? Is it something that can be fixed?

I found that the URL is mentioned in /etc/system/static/splunkrc_cmds.xml

So I tried pre-pending the port to the relevant line:
<uri><![CDATA[/shcluster/captain/kvstore-upgrade/status]]></uri>
changed to:
<uri><![CDATA[:8089/shcluster/captain/kvstore-upgrade/status]]></uri>

Restarted splunk and retried deployment, but it didn't pick up that change. So it appears that it does not use this setting in the way I thought it did. (I've reverted since that file isn't supposed to be modified).

I'm at a loss though for how to get past this issue.