Deployment Architecture
Highlighted

Why is search head cluster not showing the search head members?

New Member

I am not able to view search head members in the Members listed. I have 2 search head nodes one acting as captain (xxxxxxx02) and other acting as member. But I don't see the member listed when I run the shcluster-status (output provided below) from the member node (xxxxxx01). I am not sure if the member is part of search head clustering.

[root@xxxxx01 users]# /opt/splunk/bin/splunk show shcluster-status
In handler 'shclusterstatus': Node is not captain. Current captain = https://xxxxx02:8089

[root@xxxxxx01 users]# /opt/splunk/bin/splunk show shcluster-status

Captain:
dynamiccaptain : 1
elected
captain : Mon Apr 10 14:17:22 2017
id : 85C9FA62-9AE7-47E5-A7D6-D114C2B15BCC
initializedflag : 0
label : xxxxxxxx02
mgmt
uri : https://xxxxx02:8089
minpeersjoinedflag : 0
rolling
restartflag : 0
service
ready_flag : 0

Members:
xxxxxxx02
label : xxxxxxx02
mgmturi : https://xxxx02:8089
mgmt
uri_alias : https://xxxxx02:8089
status : Up

I get the same status output from the Captain node (xxxxx02) which doesn't show the server - xxxxx01 listed as the member. How do I confirm if the clustering is setup properly and if both nodes are under search head clustering?

0 Karma
Highlighted

Re: Why is search head cluster not showing the search head members?

SplunkTrust
SplunkTrust

The search cluster requires a minimun of 3 members. Do you have 3 nodes or you're trying with just two?

0 Karma
Highlighted

Re: Why is search head cluster not showing the search head members?

New Member

I am using only 2 search head one as Captain and one as member.

0 Karma
Highlighted

Re: Why is search head cluster not showing the search head members?

Influencer

Like @somesoni2 mentioned. You need atleast 3 members to form a cluster

Required number of instances
The cluster must contain at a minimum the number of members needed to fulfill both of these requirements:

Three members, so that the cluster can continue to function if one member goes down. See Captain election process has deployment implications.
The replication factor number of instances. See Choose the replication factor for the search head cluster.
For example, if your replication factor is either 2 or 3, you need at least three instances. If your replication factor is 5, you need at least five instances.

You can optionally add more members to boost search and user capacity.

http://docs.splunk.com/Documentation/Splunk/6.5.3/DistSearch/SHCsystemrequirements

0 Karma
Highlighted

Re: Why is search head cluster not showing the search head members?

Motivator

I have 3 node SH cluster , I want to confirm if 2 of my nodes failed - does the cluster (with the remaining 1 node) can able to accept the new search requests ?

0 Karma
Highlighted

Re: Why is search head cluster not showing the search head members?

New Member

I have now 3 nodes ( 2 Search head members and 1 Captain) but still I don't get the SHC members listed when I run the status command from the members. It just shows the Captain as shown below.

[root@muw1splmonpin01 cortana]# /opt/splunk/bin/splunk show shcluster-status

Captain:
dynamiccaptain : 1
elected
captain : Tue Apr 11 10:25:15 2017
id : 85C9FA62-9AE7-47E5-A7D6-D114C2B15BCC
initializedflag : 0
label : muw1splmonpin02
mgmt
uri : https://10.142.98.6:8089
minpeersjoinedflag : 0
rolling
restartflag : 0
service
ready_flag : 0

Members:
muw1splmonpin02
label : muw1splmonpin02
mgmturi : https://10.142.98.6:8089
mgmt
uri_alias : https://10.142.98.6:8089
status : Up

**

  • Splunkd.log from the Captain Node showing errors:

**
04-11-2017 10:30:12.530 +0000 INFO SHCMaster - event=heartbeat guid=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 msg='signaling Initial-Add (received heartbeat from Down peer)'

04-11-2017 10:30:12.559 +0000 ERROR SHCMasterPeerHandler - Cannot add peer=10.142.98.5 mgmtport=8089 (reason: removeOldPeer peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868, serverName=muw1splmonpin01, hostport=10.142.98.5:8089, but found different peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 with serverName=muw1splmonpin02 and hostport=10.142.98.6:8089 already registered and UP)

04-11-2017 10:30:16.435 +0000 INFO SHCMaster - event=heartbeat guid=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 msg='signaling Initial-Add (received heartbeat from Down peer)'

04-11-2017 10:30:16.445 +0000 ERROR SHCMasterPeerHandler - Cannot add peer=10.142.98.7 mgmtport=8089 (reason: removeOldPeer peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868, serverName=muw1splmonpin03, hostport=10.142.98.7:8089, but found different peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 with serverName=muw1splmonpin02 and hostport=10.142.98.6:8089 already registered and UP)

**

  • Splunkd log from one of the Member showing warning and error:

**

04-11-2017 10:35:38.132 +0000 INFO SHCSlave - event=SHPSlave::addPreexistingArtifacts alivesids=0 donesids=0 notdonesids(skipped)=0 artifacts=0 replicas=0
04-11-2017 10:35:38.136 +0000 WARN SHCMasterHTTPProxy - Low Level http request failure err=failed method=POST path=/services/shcluster/captain/members captain=10.142.98.6:8089 rc=0 actual
responsecode=500 expectedresponsecode=201 statusline="Internal Server Error" transaction_error="\n \n \n In handler 'shclustercaptainmembers': Cannot add peer=10.142.98.7 mgmtport=8089 (reason: removeOldPeer peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868, serverName=muw1splmonpin03, hostport=10.142.98.7:8089, but found different peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 with serverName=muw1splmonpin02 and hostport=10.142.98.6:8089 already registered and UP)\n \n\n"

04-11-2017 10:35:38.136 +0000 INFO SHCSlave - readd + haveMinPeersJoined failed err=failed method=POST path=/services/shcluster/captain/members captain=10.142.98.6:8089 rc=0 actualresponsecode=500 expectedresponsecode=201 statusline="Internal Server Error" transactionerror="\n \n \n In handler 'shclustercaptainmembers': Cannot add peer=10.142.98.7 mgmtport=8089 (reason: removeOldPeer peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868, serverName=muw1splmonpin03, hostport=10.142.98.7:8089, but found different peer=4FCA0F9D-CA10-4A83-8593-3CC0EBDB2868 with serverName=muw1splmonpin02 and hostport=10.142.98.6:8089 already registered and UP)\n \n\n" but proxy is connected. Either add-peer failed on captain, or we must be one of the early members joining the new captain

0 Karma