Deployment Architecture

How to set a new pass4SymmKey password on a search head cluster deployer?

Raghav2384
Motivator

Hello,

We have a Search head cluster in our environment and the person who set up the Deployer initially forgot the pass4SymmKey. Now , as a result, it's not letting me deploy content and throws the following message

Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://xyz.abc.com:8089 Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

Now the Cluster is running fine, but it's just that I can't deploy apps/content to the SHC members. Can I set a new password on the server.conf under the shclustering stanza (On Deployer) and add the same pass4SymmKey = new password to SHC members? Does it work, or do I need to re-initialize SHC members after adding the new password?

Appreciate your inputs...I just want to hear if you experts have an alternative before I do it the hard way 😞

Thanks,
Raghav

1 Solution

Raghav2384
Motivator

Thank you all for helping me with this...This is what worked for me

1.I added the new password to deployer and restarted splunkd
2.I initialized SHC process on each SHC member followed by a restart
./splunk init shcluster-config -auth xxx:xxx -mgmt_uri xxx:8089 -replication_port xxx -replication_factor x -conf_deploy_fetch_url xxx:8089 -secret
3.All the SHC members complain that they are not part of the cluster or yet to join (Looks scary but that message makes sense)
4. Now i can push content from deployer

Splunk docs says otherwise

Note the following:

See "Deploy a search head cluster" for details on the splunk init shcluster-config command, including the meaning of the various parameters.
The conf_deploy_fetch_url parameter specifies the URL and management port for the deployer instance. You must set it when adding a new member to an existing cluster, so that the member can immediately contact the deployer for the latest configuration bundle, if any. See "Use the deployer to distribute apps and configuration updates."
This step is for new members only. Do not run it on members rejoining the cluster.

It worked in my case. SHC and Deployer is happily married now.

Thanks,
Raghav

View solution in original post

0 Karma

Raghav2384
Motivator

Thank you all for helping me with this...This is what worked for me

1.I added the new password to deployer and restarted splunkd
2.I initialized SHC process on each SHC member followed by a restart
./splunk init shcluster-config -auth xxx:xxx -mgmt_uri xxx:8089 -replication_port xxx -replication_factor x -conf_deploy_fetch_url xxx:8089 -secret
3.All the SHC members complain that they are not part of the cluster or yet to join (Looks scary but that message makes sense)
4. Now i can push content from deployer

Splunk docs says otherwise

Note the following:

See "Deploy a search head cluster" for details on the splunk init shcluster-config command, including the meaning of the various parameters.
The conf_deploy_fetch_url parameter specifies the URL and management port for the deployer instance. You must set it when adding a new member to an existing cluster, so that the member can immediately contact the deployer for the latest configuration bundle, if any. See "Use the deployer to distribute apps and configuration updates."
This step is for new members only. Do not run it on members rejoining the cluster.

It worked in my case. SHC and Deployer is happily married now.

Thanks,
Raghav

0 Karma

ddrillic
Ultra Champion

We ended up changing the pass4SymmKey password on a Hunk SH cluster.

We followed the steps from Configure search head clustering

It came down to running the following on the deployer -
1) Changing the server.conf
2) Running the ./splunk apply shcluster-bundle command with its parameters which says -
-- Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.

Raghav2384
Motivator

Thanks for the input. Looks like the link is expired.

Question is, when you said you updated the pass4SymmKey, have you changed it on all the Search head cluster members first and then update it on deployer OR first on deployer and then the search head cluster members?

My only worry/concern here is, deployer can knock out changes made if the content under shcluster/app/* is not the same as on the SHC members.

Appreciate your help!

Thanks,
Raghav

0 Karma

ppablo
Retired

I just fixed the link, so it should work now.

0 Karma

ddrillic
Ultra Champion

We did it just on the deployer and then propagated the change via the ./splunk apply shcluster-bundle command.

0 Karma

Raghav2384
Motivator

Guess i am not lucky as you 😞

updated the pass4SymmKey on deployer's server.conf ($SPLUNK_HOME/etc/system/local/), restarted splunkd on deployer.

Once deployer is back up, ran ./splunk apply shcluster-bundle --answer-yes -target https://xyz.com:8089 -auth admin:password

I get the same error again Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://xyz.abc.com:8089 Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

Any thing else you suggest?

Thanks,
Raghav

0 Karma

ddrillic
Ultra Champion

Raghav, it appears to be an authentication error...

0 Karma

SarahBOA
Path Finder

I had the same issue, and to be clear the pass4SymmKey from the SHC and the SHCDS need to match:
The SHCDS under the [general] stanza in server.conf must match the SHC members under the [shclustering] stanza in server.conf.

It wasn't clear which stanza was needing to be updated for each and I found this to be what needed to happen.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I've not done that (updating the Pass4SymmetryKey ) myselft, but based on the documentation, I guess you can update it either
1. Using Splunk CLI
2. Updating server.conf directly.

http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/SHCconfigurationoverview#Configuration_...

Raghav2384
Motivator

Thanks Somesh.

I can update the password by adding [shclustering]pass4SymmKey under deployer for sure. My question is, looks like the password is encrypted and the person who set do not remember it. Can i delete the existing encrypted password from deployer and introduce the same password to all the SHC members and initiate a rolling-restart.

My only worry is, since deployer is capable of deleting content from SHC members, i need a little courage and words of wisdom from you guys 🙂

Appreciate all your help!

Thanks,
Raghav

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Which location on Deployer and Search Head you're making server.conf changes, etc/system/local ? If that's the case, you should be safe from deployer deleting apps from SH. As we all do before making any big change, take a backup of stuffs (etc/apps and etc/users).

Raghav2384
Motivator

Thank you Somesh....here's what i did

I updated the pass4SymmKey on deployer's server.conf ($SPLUNK_HOME/etc/system/local/), restarted splunkd on deployer.

Once deployer is back up, ran ./splunk apply shcluster-bundle --answer-yes -target https://xyz.com:8089 -auth admin:password

I get the same error again Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://xyz.abc.com:8089 Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

Any thing else you suggest?

Thanks,
Raghav

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...