Deployment Architecture
Highlighted

failure during peer node setup

Path Finder

I get the following error when I try setting up my Peer node:

Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers master=https://sjcsplunkmstr01-prod.tlcinternal.com:8089 rv=0 actualresponsecode=500 expectedresponsecode=201 statusline=HTTP/1.1 500 Internal Server Error [ event=addPeer status=retrying AddPeerRequest: { _id= activebundleid=8ABFFC208D2DCE1B6478276AA0188FC5 addtype=Initial-Add basegenerationid=0 forwarderdatarcvport=9997 forwarderdatausessl=0 mgmtport=8089 name=3A231B5A-EE5D-4F6F-A690-4198A2DE24CC registerforwarderaddress= registerreplicationaddress= registersearchaddress= replicationport=8080 replicationusessl=0 replications= server_name=sjcsplunk01-prod.tlcinternal.com status=Up } ].

On the master node, the following message appears:

Failed to add peer 'guid=3A231B5A-EE5D-4F6F-A690-4198A2DE24CC server name=sjcsplunk01-prod.tlcinternal.com ip=10.128.16.163:8089' to the master. Error=site not in available_sites

I have gone through a post which suggests checking the port status. Ports 8080 and 8089 are open.
I was able to add one node to the cluster, only the 2nd node fails. However, the failing node was acting as a stand-alone indexer before, whereas the added node was a new installation.

What could be the reason for this failure?

Tags (2)
0 Karma
Highlighted

Re: failure during peer node setup

Contributor

what version of splunk are u running?

 Error=site not in available_sites

claims that the peer does not have the right 'site' set up, if multisite is turned on, every peer must have a 'site' that is listed in the master's 'available_sites' setting (server.conf). if multisite is turned off, the peers should not have a 'site' attribute

Highlighted

Re: failure during peer node setup

Path Finder

We are using the latest version - 6.1.3 of Splunk. Multisite is turned off and peers dont have a 'site' attribute in the peer server.conf files.

0 Karma
Highlighted

Re: failure during peer node setup

Contributor

is the peer not the same version as the master?

Highlighted

Re: failure during peer node setup

Path Finder

You are right. The splunk versions were different. Having the same versions solved this problem.

Highlighted

Re: failure during peer node setup

Explorer

I am having the same issue as the above, but both master and peer nodes are on the same versions (6.2.4, build 271043). Any ideas on what is causing it?

0 Karma
Highlighted

Re: failure during peer node setup

Contributor

do you have multisite turned on, but did not set the site on the indexer? (or vice versa)

0 Karma
Highlighted

Re: failure during peer node setup

Explorer

Mulitsite is not turned on. Here are the server.conf for Master and Peer nodes:

MASTER NODE:

[sslConfig]
sslKeysfilePassword = $1$cGranKIu+bsX

[lmpool:autogeneratedpooldownload-trial]
description = auto
generatedpooldownload-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:autogeneratedpoolforwarder]
description = auto
generatedpoolforwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:autogeneratedpoolfree]
description = auto
generatedpoolfree
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = $1$JyaOwOdosukX
serverName = splunkp01
site = default

[clustering]
accessloggingforheartbeats = 1
max
peerbuildload = 5
mode = master
pass4SymmKey = $1$Nz6D2+5m7t8=
replicationfactor = 2
search
factor = 1

[lmpool:autogeneratedpoolenterprise]
description = auto
generatedpoolenterprise
quota = MAX
slaves = *
stack_id = enterprise

[license]
active_group = Enterprise

PEER NODE:

[sslConfig]
sslKeysfilePassword = $1$nG7D0aK47qLW

[lmpool:autogeneratedpooldownload-trial]
description = auto
generatedpooldownload-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:autogeneratedpoolforwarder]
description = auto
generatedpoolforwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:autogeneratedpoolfree]
description = auto
generatedpoolfree
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = $1$yyKXjef+pfDW
serverName = splunkp02
site = default

[clustering]
master_uri = https://xxx.xxx.xxx.50:8089
mode = slave
pass4SymmKey = $1$2zqalu7w+cY=

[lmpool:autogeneratedpoolenterprise]
description = auto
generatedpoolenterprise
quota = MAX
slaves = *
stack_id = enterprise

[license]
active_group = Enterprise

0 Karma
Highlighted

Re: failure during peer node setup

Builder

Yesterday I had the similar issue. The rootcause was the indexer had the 6.0.2 and Cluster Master with 6.1.3. After upgrading the indexers to 6.1.3 (same version as CM), the problem resolved.

View solution in original post