How to set a new pass4SymmKey password on a search head cluster deployer?

[Raghav2384]

Hello,

We have a Search head cluster in our environment and the person who set up the Deployer initially forgot the pass4SymmKey. Now , as a result, it's not letting me deploy content and throws the following message

Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://xyz.abc.com:8089 Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

Now the Cluster is running fine, but it's just that I can't deploy apps/content to the SHC members. Can I set a new password on the server.conf under the shclustering stanza (On Deployer) and add the same pass4SymmKey = new password to SHC members? Does it work, or do I need to re-initialize SHC members after adding the new password?

Appreciate your inputs...I just want to hear if you experts have an alternative before I do it the hard way 😞

Thanks,
Raghav

[Raghav2384 · Accepted solution]

Thank you all for helping me with this...This is what worked for me

1.I added the new password to deployer and restarted splunkd
2.I initialized SHC process on each SHC member followed by a restart
./splunk init shcluster-config -auth xxx:xxx -mgmt_uri xxx:8089 -replication_port xxx -replication_factor x -conf_deploy_fetch_url xxx:8089 -secret
3.All the SHC members complain that they are not part of the cluster or yet to join (Looks scary but that message makes sense)
4. Now i can push content from deployer

Splunk docs says otherwise

Note the following:

See "Deploy a search head cluster" for details on the splunk init shcluster-config command, including the meaning of the various parameters.
The conf_deploy_fetch_url parameter specifies the URL and management port for the deployer instance. You must set it when adding a new member to an existing cluster, so that the member can immediately contact the deployer for the latest configuration bundle, if any. See "Use the deployer to distribute apps and configuration updates."
This step is for new members only. Do not run it on members rejoining the cluster.

It worked in my case. SHC and Deployer is happily married now.

Thanks,
Raghav

[ddrillic]

We ended up changing the pass4SymmKey password on a Hunk SH cluster.

We followed the steps from Configure search head clustering

It came down to running the following on the deployer -
1) Changing the server.conf
2) Running the ./splunk apply shcluster-bundle command with its parameters which says -
-- Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.

[Raghav2384]

Thanks for the input. Looks like the link is expired.

Question is, when you said you updated the pass4SymmKey, have you changed it on all the Search head cluster members first and then update it on deployer OR first on deployer and then the search head cluster members?

My only worry/concern here is, deployer can knock out changes made if the content under shcluster/app/* is not the same as on the SHC members.

Appreciate your help!

Thanks,
Raghav

[ppablo]

I just fixed the link, so it should work now.

[ddrillic]

We did it just on the deployer and then propagated the change via the ./splunk apply shcluster-bundle command.

[Raghav2384]

Guess i am not lucky as you 😞

updated the pass4SymmKey on deployer's server.conf ($SPLUNK_HOME/etc/system/local/), restarted splunkd on deployer.

Once deployer is back up, ran ./splunk apply shcluster-bundle --answer-yes -target https://xyz.com:8089 -auth admin:password

I get the same error again Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://xyz.abc.com:8089 Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

Any thing else you suggest?

Thanks,
Raghav

[ddrillic]

Raghav, it appears to be an authentication error...

[SarahBOA]

I had the same issue, and to be clear the pass4SymmKey from the SHC and the SHCDS need to match:
The SHCDS under the [general] stanza in server.conf must match the SHC members under the [shclustering] stanza in server.conf.

It wasn't clear which stanza was needing to be updated for each and I found this to be what needed to happen.

[somesoni2]

I've not done that (updating the Pass4SymmetryKey ) myselft, but based on the documentation, I guess you can update it either
1. Using Splunk CLI
2. Updating server.conf directly.

http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/SHCconfigurationoverview#Configuration_...

[Raghav2384]

Thanks Somesh.

I can update the password by adding [shclustering]pass4SymmKey under deployer for sure. My question is, looks like the password is encrypted and the person who set do not remember it. Can i delete the existing encrypted password from deployer and introduce the same password to all the SHC members and initiate a rolling-restart.

My only worry is, since deployer is capable of deleting content from SHC members, i need a little courage and words of wisdom from you guys 🙂

Appreciate all your help!

Thanks,
Raghav

[somesoni2]

Which location on Deployer and Search Head you're making server.conf changes, etc/system/local ? If that's the case, you should be safe from deployer deleting apps from SH. As we all do before making any big change, take a backup of stuffs (etc/apps and etc/users).

[Raghav2384]

Thank you Somesh....here's what i did

I updated the pass4SymmKey on deployer's server.conf ($SPLUNK_HOME/etc/system/local/), restarted splunkd on deployer.

Once deployer is back up, ran ./splunk apply shcluster-bundle --answer-yes -target https://xyz.com:8089 -auth admin:password

I get the same error again Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://xyz.abc.com:8089 Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

Any thing else you suggest?

Thanks,
Raghav