I have now 3 nodes ( 2 Search head members and 1 Captain) but still I don't get the SHC members listed when I run the status command from the members. It just shows the Captain as shown below.
[root@muw1splmonpin01 cortana]# /opt/splunk/bin/splunk show shcluster-status
Captain:
dynamic_captain : 1
elected_captain : Tue Apr 11 10:25:15 2017
id : 85C9FA62-9AE7-47E5-A7D6-D114C2B15BCC
initialized_flag : 0
label : muw1splmonpin02
mgmt_uri : https://10.142.98.6:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0
Members:
muw1splmonpin02
label : muw1splmonpin02
mgmt_uri : https://10.142.98.6:8089
mgmt_uri_alias : https://10.142.98.6:8089
status : Up
**
Splunkd.log from the Captain Node showing errors:
**
04-11-2017 10:30:12.530 +0000 INFO SHCMaster - event=heartbeat guid=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 msg='signaling Initial-Add (received heartbeat from Down peer)'
04-11-2017 10:30:12.559 +0000 ERROR SHCMasterPeerHandler - Cannot add peer=10.142.98.5 mgmtport=8089 (reason: removeOldPeer peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868, serverName=muw1splmonpin01, hostport=10.142.98.5:8089, but found different peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 with serverName=muw1splmonpin02 and hostport=10.142.98.6:8089 already registered and UP)
04-11-2017 10:30:16.435 +0000 INFO SHCMaster - event=heartbeat guid=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 msg='signaling Initial-Add (received heartbeat from Down peer)'
04-11-2017 10:30:16.445 +0000 ERROR SHCMasterPeerHandler - Cannot add peer=10.142.98.7 mgmtport=8089 (reason: removeOldPeer peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868, serverName=muw1splmonpin03, hostport=10.142.98.7:8089, but found different peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 with serverName=muw1splmonpin02 and hostport=10.142.98.6:8089 already registered and UP)
**
Splunkd log from one of the Member showing warning and error:
**
04-11-2017 10:35:38.132 +0000 INFO SHCSlave - event=SHPSlave::addPreexistingArtifacts alive_sids=0 done_sids=0 notdone_sids(skipped)=0 artifacts=0 replicas=0
04-11-2017 10:35:38.136 +0000 WARN SHCMasterHTTPProxy - Low Level http request failure err=failed method=POST path=/services/shcluster/captain/members captain=10.142.98.6:8089 rc=0 actual_response_code=500 expected_response_code=201 status_line="Internal Server Error" transaction_error="\n \n \n In handler 'shclustercaptainmembers': Cannot add peer=10.142.98.7 mgmtport=8089 (reason: removeOldPeer peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868, serverName=muw1splmonpin03, hostport=10.142.98.7:8089, but found different peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 with serverName=muw1splmonpin02 and hostport=10.142.98.6:8089 already registered and UP)\n \n\n"
04-11-2017 10:35:38.136 +0000 INFO SHCSlave - readd + haveMinPeersJoined failed err=failed method=POST path=/services/shcluster/captain/members captain=10.142.98.6:8089 rc=0 actual_response_code=500 expected_response_code=201 status_line="Internal Server Error" transaction_error="\n \n \n In handler 'shclustercaptainmembers': Cannot add peer=10.142.98.7 mgmtport=8089 (reason: removeOldPeer peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868, serverName=muw1splmonpin03, hostport=10.142.98.7:8089, but found different peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 with serverName=muw1splmonpin02 and hostport=10.142.98.6:8089 already registered and UP)\n \n\n" but proxy is connected. Either add-peer failed on captain, or we must be one of the early members joining the new captain
... View more