I'm new in Splunk and have a test environment contains search head cluster with three Splunk 9.0.1 instances: one deployer and two search heads. If it important a Deployer also have an indexer cluster master role. This is a fresh install without any specific changes.
Output of splunk show shcluster-status --verbose:
Captain:
decommission_search_jobs_wait_secs : 180
dynamic_captain : 1
elected_captain : Tue Jan 24 17:57:01 2023
id : 17B17CF3-57A4-4F34-A943-835219C2DA41
initialized_flag : 1
kvstore_maintenance_status : disabled
label : spl-sh02
max_failures_to_keep_majority : 0
mgmt_uri : https://spl-sh02.domain.com:8089
min_peers_joined_flag : 1
rolling_restart : restart
rolling_restart_flag : 0
rolling_upgrade_flag : 0
service_ready_flag : 1
stable_captain : 1
Cluster Manager(s):
https://spl-ms01.domain.com:8089 splunk_version: 9.0.0.1
Members:
spl-sh02
kvstore_status : ready
label : spl-sh02
manual_detention : off
mgmt_uri : https://domain.com:8089
mgmt_uri_alias : https://172.28.56.104:8089
out_of_sync_node : 0
preferred_captain : 1
restart_required : 0
splunk_version : 9.0.0.1
status : Up
spl-sh01
kvstore_status : ready
label : spl-sh01
last_conf_replication : Wed Jan 25 10:52:26 2023
manual_detention : off
mgmt_uri : https://spl-sh01.domain.com:8089
mgmt_uri_alias : https://172.28.56.100:8089
out_of_sync_node : 0
preferred_captain : 1
restart_required : 0
splunk_version : 9.0.0.1
status : Up
When i'm try to execute "apply shcluster-bundle" on deployer i'm see this error:
Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Error in pre-deploy check, uri=https://spl-sh02.domain.com:8089/services/shcluster/captain/kvstore-upgrade/status, status=401, error=No error
How i can solve this problem?
I was find resolution by myself.
As most of us knows error 401 typically means 'invalid authentication credentials'.
This sounds pretty clear but i didn't pay attention about it for a long time because i'm using Ansible and thought that passwors on Deployer and SHC members (SHCms) can't mismatch.
But i was wrong.
Due to installation peculiarityes a password for Deployer ans SHCms seting by different ways. On the one hand you need to set password directly in Deployer's server.conf. On the other hand you need to use 'splunk init shcluster-config' command to create SHC.
My ansible code for SHC creating executed by the win_shell module and looked like:
- name: 'SPL :: Configuring Search head cluster member'
#ansible.windows.win_shell
win_shell: |
splunk init shcluster-config -auth '{{ admin }}:{{ password }}' -mgmt_uri '{{ mgmt_uri}}' ... bla bla bla ... -secret '{{ shc_key }}' -shcluster_label '{{ shc_label }}'
When i was use this code i had password mismatch on Deployer and SHCms.
As an experiment i ran command without secret argument then i set seсret directly in server.conf... And it worked!
Based on this i think that 'splunk init shcluster-config' command have some errors and don't ignoring single quotes which need to escaping chars when you use ansible.
Other commands e.g. 'splunk edit cluster-config' or 'splunk edit cluster-config' don't have this error.
I was find resolution by myself.
As most of us knows error 401 typically means 'invalid authentication credentials'.
This sounds pretty clear but i didn't pay attention about it for a long time because i'm using Ansible and thought that passwors on Deployer and SHC members (SHCms) can't mismatch.
But i was wrong.
Due to installation peculiarityes a password for Deployer ans SHCms seting by different ways. On the one hand you need to set password directly in Deployer's server.conf. On the other hand you need to use 'splunk init shcluster-config' command to create SHC.
My ansible code for SHC creating executed by the win_shell module and looked like:
- name: 'SPL :: Configuring Search head cluster member'
#ansible.windows.win_shell
win_shell: |
splunk init shcluster-config -auth '{{ admin }}:{{ password }}' -mgmt_uri '{{ mgmt_uri}}' ... bla bla bla ... -secret '{{ shc_key }}' -shcluster_label '{{ shc_label }}'
When i was use this code i had password mismatch on Deployer and SHCms.
As an experiment i ran command without secret argument then i set seсret directly in server.conf... And it worked!
Based on this i think that 'splunk init shcluster-config' command have some errors and don't ignoring single quotes which need to escaping chars when you use ansible.
Other commands e.g. 'splunk edit cluster-config' or 'splunk edit cluster-config' don't have this error.