Installation

How to recover the Pass4SymmKey after moving the deployer to a new host?

Glasses2
Path Finder

After following, well verified steps as noted in > https://community.splunk.com/t5/Deployment-Architecture/How-to-move-the-SHC-deployer-to-another-host...

I was not able to successfully connect and test a push from the new deployer to the shcluster members.  I received an error >>> Error while deploying apps to first member, aborting apps deployment to all members: Error while fetching apps baseline on target=https://host:8089: Non-200/201 status_code=401; {"messages":[{"type":"ERROR","text":"Unauthorized"}]}

Here are my steps:
1. copied the contents of /opt/splunk/etc/shcluster from the old deployer to the new deployer /opt/splunk/etc/shcluster
2) configured the new deployer [shclustering] stanza with the info from the old deployer [shclustering] stanza in /opt/splunk/etc/system/local server.conf
3) Updated conf_deploy_fetch_url in server.conf on each of the shc members
4) restarted the new deployer and a rolling restart on the shc members
5) did a test apply bundle and then received an error unauthorized.

I believe the issue could be with the pass4SymmKey (on the new deployer) not being the same as the pass4SymmKey on the SHC members.

I did a ./splunk show-decrypt --value <key> from the old deployer
[shclustering]
pass4SymmKey = <key>
shcluster_label = Company_shcluster1

I used the decrypted key as the key for the new deployer pass3SymmKey but ultimately I am not able to run a successful push.

Is there a way to recover these keys? The previous admin did not save the original secrets used to setup the deployer.

Any advice greatly appreciated.

Thank you

Labels (1)
0 Karma
1 Solution

Glasses2
Path Finder

I actually was able to find the original secret that was used, so I got lucky.

After configuring the conf with the secret and a restart, test push was successful.

But thank you for the pointers, much appreciated!

View solution in original post

0 Karma

isoutamo
SplunkTrust
SplunkTrust

When you are running 

splunk show shcluster-status --verbose
splunk show kvstore-status --verbose

On those SHC nodes will you get a successful status with healthy SHC and KVStore?

Which version you have and which OS?

Easy to say after your issue, that I always prefer to use FQDN (usually CNAMEs) for all nodes to avoid this kind of issues when switch to another deployer, CM, LM or other....

r. Ismo 

Glasses2
Path Finder

I actually was able to find the original secret that was used, so I got lucky.

After configuring the conf with the secret and a restart, test push was successful.

But thank you for the pointers, much appreciated!

0 Karma

isoutamo
SplunkTrust
SplunkTrust
When you run that “splunk show-decrypt …” on original deployer it should show the original key. But if you are running it on some other nodes with server.conf from original deployer and random splunk.secret file then it shows random text to you.

Glasses2
Path Finder

Thank you for the additional info.

That is the weird part, I  was running the show-decrypted on the original and I was getting garbage.  

 

I just tried the show-decrypted again on the shclustering pass4SymmKey and I get just an "="  so I am not sure what is happening... but the password works...

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...