topic Error while applying shcluster-bundle on deployer? in Deployment Architecture

Error while applying shcluster-bundle on deployer?

NoSpaces — Wed, 25 Jan 2023 16:47:26 GMT

I'm new in Splunk and have a test environment contains search head cluster with three Splunk 9.0.1 instances: one deployer and two search heads. If it important a Deployer also have an indexer cluster master role. This is a fresh install without any specific changes.
Output of splunk show shcluster-status --verbose:

Captain: decommission_search_jobs_wait_secs : 180 dynamic_captain : 1 elected_captain : Tue Jan 24 17:57:01 2023 id : 17B17CF3-57A4-4F34-A943-835219C2DA41 initialized_flag : 1 kvstore_maintenance_status : disabled label : spl-sh02 max_failures_to_keep_majority : 0 mgmt_uri : https://spl-sh02.domain.com:8089 min_peers_joined_flag : 1 rolling_restart : restart rolling_restart_flag : 0 rolling_upgrade_flag : 0 service_ready_flag : 1 stable_captain : 1 Cluster Manager(s): https://spl-ms01.domain.com:8089 splunk_version: 9.0.0.1 Members: spl-sh02 kvstore_status : ready label : spl-sh02 manual_detention : off mgmt_uri : https://domain.com:8089 mgmt_uri_alias : https://172.28.56.104:8089 out_of_sync_node : 0 preferred_captain : 1 restart_required : 0 splunk_version : 9.0.0.1 status : Up spl-sh01 kvstore_status : ready label : spl-sh01 last_conf_replication : Wed Jan 25 10:52:26 2023 manual_detention : off mgmt_uri : https://spl-sh01.domain.com:8089 mgmt_uri_alias : https://172.28.56.100:8089 out_of_sync_node : 0 preferred_captain : 1 restart_required : 0 splunk_version : 9.0.0.1 status : Up

When i'm try to execute "apply shcluster-bundle" on deployer i'm see this error:

Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. Error in pre-deploy check, uri=https://spl-sh02.domain.com:8089/services/shcluster/captain/kvstore-upgrade/status, status=401, error=No error

How i can solve this problem?

Re: Error while applying shcluster-bundle on deployer?

NoSpaces — Fri, 03 Feb 2023 07:26:33 GMT

I was find resolution by myself.
As most of us knows error 401 typically means 'invalid authentication credentials'.
This sounds pretty clear but i didn't pay attention about it for a long time because i'm using Ansible and thought that passwors on Deployer and SHC members (SHCms) can't mismatch.
But i was wrong.
Due to installation peculiarityes a password for Deployer ans SHCms seting by different ways. On the one hand you need to set password directly in Deployer's server.conf. On the other hand you need to use 'splunk init shcluster-config' command to create SHC.
My ansible code for SHC creating executed by the win_shell module and looked like:

- name: 'SPL :: Configuring Search head cluster member' #ansible.windows.win_shell win_shell: | splunk init shcluster-config -auth '{{ admin }}:{{ password }}' -mgmt_uri '{{ mgmt_uri}}' ... bla bla bla ... -secret '{{ shc_key }}' -shcluster_label '{{ shc_label }}'

When i was use this code i had password mismatch on Deployer and SHCms.

As an experiment i ran command without secret argument then i set seсret directly in server.conf... And it worked!

Based on this i think that 'splunk init shcluster-config' command have some errors and don't ignoring single quotes which need to escaping chars when you use ansible.
Other commands e.g. 'splunk edit cluster-config' or 'splunk edit cluster-config' don't have this error.

Re: Error while applying shcluster-bundle on deployer?

splunkreal — Fri, 07 Mar 2025 15:26:15 GMT

Hello @NoSpaces did you report to support as this looks like splunk init shcluster-config bug?