Deployment Architecture

failure during peer node setup

nivedita_viswan
Path Finder

I get the following error when I try setting up my Peer node:

Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers master=https://sjcsplunkmstr01-prod.tlcinternal.com:8089 rv=0 actual_response_code=500 expected_response_code=201 status_line=HTTP/1.1 500 Internal Server Error [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=8ABFFC208D2DCE1B6478276AA0188FC5 add_type=Initial-Add base_generation_id=0 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 mgmt_port=8089 name=3A231B5A-EE5D-4F6F-A690-4198A2DE24CC register_forwarder_address= register_replication_address= register_search_address= replication_port=8080 replication_use_ssl=0 replications= server_name=sjcsplunk01-prod.tlcinternal.com status=Up } ].

On the master node, the following message appears:

Failed to add peer 'guid=3A231B5A-EE5D-4F6F-A690-4198A2DE24CC server name=sjcsplunk01-prod.tlcinternal.com ip=10.128.16.163:8089' to the master. Error=site not in available_sites

I have gone through a post which suggests checking the port status. Ports 8080 and 8089 are open.
I was able to add one node to the cluster, only the 2nd node fails. However, the failing node was acting as a stand-alone indexer before, whereas the added node was a new installation.

What could be the reason for this failure?

Tags (2)
0 Karma
1 Solution

jayannah
Builder

Yesterday I had the similar issue. The rootcause was the indexer had the 6.0.2 and Cluster Master with 6.1.3. After upgrading the indexers to 6.1.3 (same version as CM), the problem resolved.

View solution in original post

jayannah
Builder

Yesterday I had the similar issue. The rootcause was the indexer had the 6.0.2 and Cluster Master with 6.1.3. After upgrading the indexers to 6.1.3 (same version as CM), the problem resolved.

dxu_splunk
Splunk Employee
Splunk Employee

what version of splunk are u running?

 Error=site not in available_sites

claims that the peer does not have the right 'site' set up, if multisite is turned on, every peer must have a 'site' that is listed in the master's 'available_sites' setting (server.conf). if multisite is turned off, the peers should not have a 'site' attribute

z5pate
Explorer

I am having the same issue as the above, but both master and peer nodes are on the same versions (6.2.4, build 271043). Any ideas on what is causing it?

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

do you have multisite turned on, but did not set the site on the indexer? (or vice versa)

0 Karma

z5pate
Explorer

Mulitsite is not turned on. Here are the server.conf for Master and Peer nodes:

MASTER NODE:

[sslConfig]
sslKeysfilePassword = $1$cGranKIu+bsX

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = $1$JyaOwOdosukX
serverName = splunkp01
site = default

[clustering]
access_logging_for_heartbeats = 1
max_peer_build_load = 5
mode = master
pass4SymmKey = $1$Nz6D2+5m7t8=
replication_factor = 2
search_factor = 1

[lmpool:auto_generated_pool_enterprise]
description = auto_generated_pool_enterprise
quota = MAX
slaves = *
stack_id = enterprise

[license]
active_group = Enterprise

PEER NODE:

[sslConfig]
sslKeysfilePassword = $1$nG7D0aK47qLW

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = $1$yyKXjef+pfDW
serverName = splunkp02
site = default

[clustering]
master_uri = https://xxx.xxx.xxx.50:8089
mode = slave
pass4SymmKey = $1$2zqalu7w+cY=

[lmpool:auto_generated_pool_enterprise]
description = auto_generated_pool_enterprise
quota = MAX
slaves = *
stack_id = enterprise

[license]
active_group = Enterprise

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

is the peer not the same version as the master?

nivedita_viswan
Path Finder

You are right. The splunk versions were different. Having the same versions solved this problem.

nivedita_viswan
Path Finder

We are using the latest version - 6.1.3 of Splunk. Multisite is turned off and peers dont have a 'site' attribute in the peer server.conf files.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...