How to recover the Pass4SymmKey after moving the deployer to a new host?

[Glasses2]

After following, well verified steps as noted in > https://community.splunk.com/t5/Deployment-Architecture/How-to-move-the-SHC-deployer-to-another-host...

I was not able to successfully connect and test a push from the new deployer to the shcluster members.  I received an error >>> Error while deploying apps to first member, aborting apps deployment to all members: Error while fetching apps baseline on target=https://host:8089: Non-200/201 status_code=401; {"messages":[{"type":"ERROR","text":"Unauthorized"}]}

Here are my steps:
1. copied the contents of /opt/splunk/etc/shcluster from the old deployer to the new deployer /opt/splunk/etc/shcluster
2) configured the new deployer [shclustering] stanza with the info from the old deployer [shclustering] stanza in /opt/splunk/etc/system/local server.conf
3) Updated conf_deploy_fetch_url in server.conf on each of the shc members
4) restarted the new deployer and a rolling restart on the shc members
5) did a test apply bundle and then received an error unauthorized.

I believe the issue could be with the pass4SymmKey (on the new deployer) not being the same as the pass4SymmKey on the SHC members.

I did a ./splunk show-decrypt --value <key> from the old deployer
[shclustering]
pass4SymmKey = <key>
shcluster_label = Company_shcluster1

I used the decrypted key as the key for the new deployer pass3SymmKey but ultimately I am not able to run a successful push.

Is there a way to recover these keys? The previous admin did not save the original secrets used to setup the deployer.

Any advice greatly appreciated.

Thank you

[isoutamo]

When you are running 

splunk show shcluster-status --verbose
splunk show kvstore-status --verbose

On those SHC nodes will you get a successful status with healthy SHC and KVStore?

Which version you have and which OS?

Easy to say after your issue, that I always prefer to use FQDN (usually CNAMEs) for all nodes to avoid this kind of issues when switch to another deployer, CM, LM or other....

r. Ismo 

[Glasses2 · Accepted solution]

I actually was able to find the original secret that was used, so I got lucky.

After configuring the conf with the secret and a restart, test push was successful.

But thank you for the pointers, much appreciated!

[isoutamo]

When you run that “splunk show-decrypt …” on original deployer it should show the original key. But if you are running it on some other nodes with server.conf from original deployer and random splunk.secret file then it shows random text to you.

[Glasses2]

Thank you for the additional info.

That is the weird part, I  was running the show-decrypted on the original and I was getting garbage.  

I just tried the show-decrypted again on the shclustering pass4SymmKey and I get just an "="  so I am not sure what is happening... but the password works...