topic Re: Why "show shcluster-status" only shows the captain in Members list? in Splunk Search

Why "show shcluster-status" only shows the captain in Members list?

guotao4321 — Tue, 29 Sep 2020 11:30:28 GMT

I setup a search head cluster on 3 search heads:

[root@deploy-searchhead01 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead01.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!

[root@deploy-searchhead02 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead02.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!

[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead03.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!

I bootstrapped searchhead03 as the captain with the 3 search head:

[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://deploy-searchhead01.bigdata.emc.local:8089,https://deploy-searchhead02.bigdata.emc.local:8089,https://deploy-searchhead03.bigdata.emc.local:8089" -auth admin:changeme

When I run the command "splunk show shcluster-status", only the captain is shown in the Members:

[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk show shcluster-status -auth admin:changeme

Captain:
dynamic_captain : 1
elected_captain : Thu Oct 20 17:35:52 2016
id : 54B57E56-AB72-4569-AE07-A5B8049C5690
initialized_flag : 0
label : deploy-searchhead03.bigdata.emc.local
mgmt_uri : https://deploy-searchhead03.bigdata.emc.local:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0

Members:
deploy-searchhead03.bigdata.emc.local
label : deploy-searchhead03.bigdata.emc.local
mgmt_uri : https://deploy-searchhead03.bigdata.emc.local:8089
mgmt_uri_alias : https://172.16.1.78:8089
status : Up

The output is same on all the 3 search heads. Please help and let me know what's wrong. The Splunk version is 6.5.0.
Thank you.

Re: Why "show shcluster-status" only shows the captain in Members list?

guotao4321 — Fri, 21 Oct 2016 03:23:55 GMT

As I used the different password for -secret with the instance password "changeme"
I can see in the server.conf file, the pass4SymmKey = changeme is under [general]. However, under [shclustering] there is no pass4SymmKey configuration..

Re: Why "show shcluster-status" only shows the captain in Members list?

guotao4321 — Tue, 29 Sep 2020 11:31:11 GMT

I configure the server.conf and add the pass4SymmKey = changeme under [shclustering] and restart the splunkd.

I rebuild the SH cluster using the -secret changeme and -shcluster_label shcluster1, however, the output is same, no other members shown in the member list:

Captain:
dynamic_captain : 1
elected_captain : Fri Oct 21 12:20:43 2016
id : 6F21DCB3-0E06-4AA0-8E1C-7FE2D2712588
initialized_flag : 0
label : deploy-searchhead01.bigdata.emc.local
mgmt_uri : https://deploy-searchhead01.bigdata.emc.local:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0

Members:
deploy-searchhead01.bigdata.emc.local
label : deploy-searchhead01.bigdata.emc.local
mgmt_uri : https://deploy-searchhead01.bigdata.emc.local:8089
mgmt_uri_alias : https://172.16.1.72:8089
status : Up

Re: Why "show shcluster-status" only shows the captain in Members list?

maciep — Sat, 22 Oct 2016 12:30:00 GMT

have you looked in the /opt/splunk/var/log/splunkd.log on the captain or members to see if there are any errors/warnings in there around running those commands or communication between servers?

Is SSL enabled by default for the management port? No idea, but was wondering if http/https was in play here at all.

Re: Why "show shcluster-status" only shows the captain in Members list?

guotao4321 — Mon, 24 Oct 2016 08:38:14 GMT

issue fixed. Check the log for reason: splunk/var/log/splunk/splunkd.log

Re: Why "show shcluster-status" only shows the captain in Members list?

maciep — Mon, 24 Oct 2016 11:58:20 GMT

Glad you got it resolved! But could you share what you found and how you fixed the issue? Somebody may stumble across this post in the future, and maybe your experience will be able to help them.

Re: Why "show shcluster-status" only shows the captain in Members list?

guotao4321 — Tue, 25 Oct 2016 01:53:04 GMT

Check the log for reason: splunk/var/log/splunk/splunkd.log
It will show the detial event why the other members are not added into the cluster.

In my case, as I cloned the Splunk search head VM, all the search heads' guids are same. That's why they cannot be added into one cluster.

Resolution:
rm /opt/splunk/etc/instance.cfg
splunk restart