Deployment Architecture

Error while applying shcluster-bundle on deployer?

NoSpaces
Engager

I'm new in Splunk and have a test environment contains search head cluster with three Splunk 9.0.1 instances: one deployer and two search heads. If it important a Deployer also have an indexer cluster master role. This is a fresh install without any specific changes. 
Output of splunk show shcluster-status --verbose:

 

 Captain:
                decommission_search_jobs_wait_secs : 180
                               dynamic_captain : 1
                               elected_captain : Tue Jan 24 17:57:01 2023
                                            id : 17B17CF3-57A4-4F34-A943-835219C2DA41
                              initialized_flag : 1
                    kvstore_maintenance_status : disabled
                                         label : spl-sh02
                 max_failures_to_keep_majority : 0
                                      mgmt_uri : https://spl-sh02.domain.com:8089
                         min_peers_joined_flag : 1
                               rolling_restart : restart
                          rolling_restart_flag : 0
                          rolling_upgrade_flag : 0
                            service_ready_flag : 1
                                stable_captain : 1

 Cluster Manager(s):
        https://spl-ms01.domain.com:8089         splunk_version: 9.0.0.1

 Members:
        spl-sh02
                                kvstore_status : ready
                                         label : spl-sh02
                              manual_detention : off
                                      mgmt_uri : https://domain.com:8089
                                mgmt_uri_alias : https://172.28.56.104:8089
                              out_of_sync_node : 0
                             preferred_captain : 1
                              restart_required : 0
                                splunk_version : 9.0.0.1
                                        status : Up
        spl-sh01
                                kvstore_status : ready
                                         label : spl-sh01
                         last_conf_replication : Wed Jan 25 10:52:26 2023
                              manual_detention : off
                                      mgmt_uri : https://spl-sh01.domain.com:8089
                                mgmt_uri_alias : https://172.28.56.100:8089
                              out_of_sync_node : 0
                             preferred_captain : 1
                              restart_required : 0
                                splunk_version : 9.0.0.1
                                        status : Up

 

When i'm try to execute "apply shcluster-bundle" on deployer i'm see this error:

 

Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.  Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Error in pre-deploy check, uri=https://spl-sh02.domain.com:8089/services/shcluster/captain/kvstore-upgrade/status, status=401, error=No error

 

How i can solve this problem? 

Labels (2)
0 Karma
1 Solution

NoSpaces
Engager

I was find resolution by myself.
As most of us knows error 401 typically means 'invalid authentication credentials'.
This sounds pretty clear but i didn't pay attention about it for a long time because i'm using Ansible and thought that passwors on Deployer and SHC members (SHCms) can't mismatch.
But i was wrong.
Due to installation peculiarityes a password for Deployer ans SHCms seting by different ways. On the one hand you need to set password directly in Deployer's server.conf. On the other hand you need to use 'splunk init shcluster-config' command to create SHC.
My ansible code for SHC creating executed by the win_shell module and looked like:

- name: 'SPL :: Configuring Search head cluster member'
  #ansible.windows.win_shell
  win_shell: |
    splunk init shcluster-config -auth '{{ admin }}:{{ password }}' -mgmt_uri '{{ mgmt_uri}}' ... bla bla bla ... -secret '{{ shc_key }}' -shcluster_label '{{ shc_label }}'

When i was use this code i had password mismatch on Deployer and SHCms.

As an experiment i ran command without secret argument then i set seсret directly in server.conf... And it worked!

Based on this i think that 'splunk init shcluster-config' command have some errors and don't ignoring single quotes which need to escaping chars when you use ansible.
Other commands e.g. 'splunk edit cluster-config' or 'splunk edit cluster-config' don't have this error.

View solution in original post

NoSpaces
Engager

I was find resolution by myself.
As most of us knows error 401 typically means 'invalid authentication credentials'.
This sounds pretty clear but i didn't pay attention about it for a long time because i'm using Ansible and thought that passwors on Deployer and SHC members (SHCms) can't mismatch.
But i was wrong.
Due to installation peculiarityes a password for Deployer ans SHCms seting by different ways. On the one hand you need to set password directly in Deployer's server.conf. On the other hand you need to use 'splunk init shcluster-config' command to create SHC.
My ansible code for SHC creating executed by the win_shell module and looked like:

- name: 'SPL :: Configuring Search head cluster member'
  #ansible.windows.win_shell
  win_shell: |
    splunk init shcluster-config -auth '{{ admin }}:{{ password }}' -mgmt_uri '{{ mgmt_uri}}' ... bla bla bla ... -secret '{{ shc_key }}' -shcluster_label '{{ shc_label }}'

When i was use this code i had password mismatch on Deployer and SHCms.

As an experiment i ran command without secret argument then i set seсret directly in server.conf... And it worked!

Based on this i think that 'splunk init shcluster-config' command have some errors and don't ignoring single quotes which need to escaping chars when you use ansible.
Other commands e.g. 'splunk edit cluster-config' or 'splunk edit cluster-config' don't have this error.

Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...