Deployment Architecture

Error while applying shcluster-bundle on deployer?

NoSpaces
Engager

I'm new in Splunk and have a test environment contains search head cluster with three Splunk 9.0.1 instances: one deployer and two search heads. If it important a Deployer also have an indexer cluster master role. This is a fresh install without any specific changes. 
Output of splunk show shcluster-status --verbose:

 

 Captain:
                decommission_search_jobs_wait_secs : 180
                               dynamic_captain : 1
                               elected_captain : Tue Jan 24 17:57:01 2023
                                            id : 17B17CF3-57A4-4F34-A943-835219C2DA41
                              initialized_flag : 1
                    kvstore_maintenance_status : disabled
                                         label : spl-sh02
                 max_failures_to_keep_majority : 0
                                      mgmt_uri : https://spl-sh02.domain.com:8089
                         min_peers_joined_flag : 1
                               rolling_restart : restart
                          rolling_restart_flag : 0
                          rolling_upgrade_flag : 0
                            service_ready_flag : 1
                                stable_captain : 1

 Cluster Manager(s):
        https://spl-ms01.domain.com:8089         splunk_version: 9.0.0.1

 Members:
        spl-sh02
                                kvstore_status : ready
                                         label : spl-sh02
                              manual_detention : off
                                      mgmt_uri : https://domain.com:8089
                                mgmt_uri_alias : https://172.28.56.104:8089
                              out_of_sync_node : 0
                             preferred_captain : 1
                              restart_required : 0
                                splunk_version : 9.0.0.1
                                        status : Up
        spl-sh01
                                kvstore_status : ready
                                         label : spl-sh01
                         last_conf_replication : Wed Jan 25 10:52:26 2023
                              manual_detention : off
                                      mgmt_uri : https://spl-sh01.domain.com:8089
                                mgmt_uri_alias : https://172.28.56.100:8089
                              out_of_sync_node : 0
                             preferred_captain : 1
                              restart_required : 0
                                splunk_version : 9.0.0.1
                                        status : Up

 

When i'm try to execute "apply shcluster-bundle" on deployer i'm see this error:

 

Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.  Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Error in pre-deploy check, uri=https://spl-sh02.domain.com:8089/services/shcluster/captain/kvstore-upgrade/status, status=401, error=No error

 

How i can solve this problem? 

Labels (2)
0 Karma
1 Solution

NoSpaces
Engager

I was find resolution by myself.
As most of us knows error 401 typically means 'invalid authentication credentials'.
This sounds pretty clear but i didn't pay attention about it for a long time because i'm using Ansible and thought that passwors on Deployer and SHC members (SHCms) can't mismatch.
But i was wrong.
Due to installation peculiarityes a password for Deployer ans SHCms seting by different ways. On the one hand you need to set password directly in Deployer's server.conf. On the other hand you need to use 'splunk init shcluster-config' command to create SHC.
My ansible code for SHC creating executed by the win_shell module and looked like:

- name: 'SPL :: Configuring Search head cluster member'
  #ansible.windows.win_shell
  win_shell: |
    splunk init shcluster-config -auth '{{ admin }}:{{ password }}' -mgmt_uri '{{ mgmt_uri}}' ... bla bla bla ... -secret '{{ shc_key }}' -shcluster_label '{{ shc_label }}'

When i was use this code i had password mismatch on Deployer and SHCms.

As an experiment i ran command without secret argument then i set seсret directly in server.conf... And it worked!

Based on this i think that 'splunk init shcluster-config' command have some errors and don't ignoring single quotes which need to escaping chars when you use ansible.
Other commands e.g. 'splunk edit cluster-config' or 'splunk edit cluster-config' don't have this error.

View solution in original post

NoSpaces
Engager

I was find resolution by myself.
As most of us knows error 401 typically means 'invalid authentication credentials'.
This sounds pretty clear but i didn't pay attention about it for a long time because i'm using Ansible and thought that passwors on Deployer and SHC members (SHCms) can't mismatch.
But i was wrong.
Due to installation peculiarityes a password for Deployer ans SHCms seting by different ways. On the one hand you need to set password directly in Deployer's server.conf. On the other hand you need to use 'splunk init shcluster-config' command to create SHC.
My ansible code for SHC creating executed by the win_shell module and looked like:

- name: 'SPL :: Configuring Search head cluster member'
  #ansible.windows.win_shell
  win_shell: |
    splunk init shcluster-config -auth '{{ admin }}:{{ password }}' -mgmt_uri '{{ mgmt_uri}}' ... bla bla bla ... -secret '{{ shc_key }}' -shcluster_label '{{ shc_label }}'

When i was use this code i had password mismatch on Deployer and SHCms.

As an experiment i ran command without secret argument then i set seсret directly in server.conf... And it worked!

Based on this i think that 'splunk init shcluster-config' command have some errors and don't ignoring single quotes which need to escaping chars when you use ansible.
Other commands e.g. 'splunk edit cluster-config' or 'splunk edit cluster-config' don't have this error.

Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...