Archive

Why "show shcluster-status" only shows the captain in Members list?

Path Finder

I setup a search head cluster on 3 search heads:

[root@deploy-searchhead01 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead01.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!

[root@deploy-searchhead02 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead02.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!

[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead03.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!

I bootstrapped searchhead03 as the captain with the 3 search head:

[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://deploy-searchhead01.bigdata.emc.local:8089,https://deploy-searchhead02.bigdata.emc.local:808..." -auth admin:changeme

When I run the command "splunk show shcluster-status", only the captain is shown in the Members:

[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk show shcluster-status -auth admin:changeme

Captain:
dynamic_captain : 1
elected_captain : Thu Oct 20 17:35:52 2016
id : 54B57E56-AB72-4569-AE07-A5B8049C5690
initialized_flag : 0
label : deploy-searchhead03.bigdata.emc.local
mgmt_uri : https://deploy-searchhead03.bigdata.emc.local:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0

Members:
deploy-searchhead03.bigdata.emc.local
label : deploy-searchhead03.bigdata.emc.local
mgmt_uri : https://deploy-searchhead03.bigdata.emc.local:8089
mgmt_uri_alias : https://172.16.1.78:8089
status : Up

The output is same on all the 3 search heads. Please help and let me know what's wrong. The Splunk version is 6.5.0.
Thank you.

Tags (1)
1 Solution

Path Finder

issue fixed. Check the log for reason: splunk/var/log/splunk/splunkd.log

View solution in original post

Path Finder

issue fixed. Check the log for reason: splunk/var/log/splunk/splunkd.log

View solution in original post

Path Finder

Check the log for reason: splunk/var/log/splunk/splunkd.log
It will show the detial event why the other members are not added into the cluster.

In my case, as I cloned the Splunk search head VM, all the search heads' guids are same. That's why they cannot be added into one cluster.

Resolution:
rm /opt/splunk/etc/instance.cfg
splunk restart

0 Karma

Champion

Glad you got it resolved! But could you share what you found and how you fixed the issue? Somebody may stumble across this post in the future, and maybe your experience will be able to help them.

0 Karma

Path Finder

I configure the server.conf and add the pass4SymmKey = changeme under [shclustering] and restart the splunkd.

I rebuild the SH cluster using the -secret changeme and -shcluster_label shcluster1, however, the output is same, no other members shown in the member list:

Captain:
dynamic_captain : 1
elected_captain : Fri Oct 21 12:20:43 2016
id : 6F21DCB3-0E06-4AA0-8E1C-7FE2D2712588
initialized_flag : 0
label : deploy-searchhead01.bigdata.emc.local
mgmt_uri : https://deploy-searchhead01.bigdata.emc.local:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0

Members:
deploy-searchhead01.bigdata.emc.local
label : deploy-searchhead01.bigdata.emc.local
mgmt_uri : https://deploy-searchhead01.bigdata.emc.local:8089
mgmt_uri_alias : https://172.16.1.72:8089
status : Up

0 Karma

Champion

have you looked in the /opt/splunk/var/log/splunkd.log on the captain or members to see if there are any errors/warnings in there around running those commands or communication between servers?

Is SSL enabled by default for the management port? No idea, but was wondering if http/https was in play here at all.

0 Karma

Path Finder

As I used the different password for -secret with the instance password "changeme"
I can see in the server.conf file, the pass4SymmKey = changeme is under [general]. However, under [shclustering] there is no pass4SymmKey configuration..

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!