Deployment Architecture

How to resolve error "Waiting for requisite number of peers to join the cluster... Cluster has only 0 peers (waiting for 3 peers to join the cluster)"?

andrewdwest
Explorer

Splunk Answers,

I started receiving this message on August 23rd, 2016. I am not sure why the peers will not join cluster?

Waiting for requisite number of peers to join the cluster - https://127.0.0.1:8089. Cluster has only 0 peers (waiting for 3 peers to join the cluster)

Please advise how to resolve the issue, thank you in advance.

1 Solution

andrewdwest
Explorer

The message/problem was being caused by a setting under "Settings" -> "Distributed Environment" -> "Distributed Search" -> "Distributed search set up" -> Change the setting from "Yes" to "No."

View solution in original post

gopmister
Explorer

Andrewwest,

Is this on the CM or indexers? I have the same issue, but I am not able to login to the CM via GUI so I am having a hard time figuring it out. Let me know where to change it.

0 Karma

andrewdwest
Explorer

The message/problem was being caused by a setting under "Settings" -> "Distributed Environment" -> "Distributed Search" -> "Distributed search set up" -> Change the setting from "Yes" to "No."

vgollapudi
Communicator

It's not necessary to disable to fix the issue. The issue can be related to the communication between the Indexer and Cluster Master. It can also occur when the Splunk service isn't running on the Indexers.

0 Karma

mrtolu6
Path Finder

so the indexer connects for about 10 sec then goes down. What are some good troubleshoot steps to help me resolve this?

0 Karma

vgollapudi
Communicator

First step is to check the logs on the Cluster Master as well as on the Indexers.

Second step is apply the cluster bundle from Cluster Master to the Indexers to ensure the configuration regarding the apps is synchronous among the peers.

Third step, whenever the indexer drops the connection check the splunk service status.

0 Karma

andrewdwest
Explorer
08-24-2016 14:31:59.193 -0700 INFO  CMMaster - event=service status=skipping reason='Cluster has only 0 peers (waiting for 1 peers to join the cluster)'

08-24-2016 14:52:50.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:52:55.636 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:00.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:05.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:10.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:15.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:20.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:25.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:30.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:30.785 -0700 INFO  CMMaster - event=service status=skipping reason='Cluster has only 0 peers (waiting for 1 peers to join the cluster)'
08-24-2016 14:53:31.852 -0700 INFO  DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/Applications/Splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 .  Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
08-24-2016 14:53:31.861 -0700 INFO  DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/Applications/Splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 .  Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
08-24-2016 14:53:35.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:40.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:45.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:50.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:55.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:00.640 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:05.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:10.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:15.640 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:20.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:25.640 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:30.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:35.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:40.636 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:45.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:50.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:55.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:00.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:05.636 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:10.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:15.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:20.640 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:25.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:30.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:35.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:40.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:45.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:50.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:55.634 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:00.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:05.634 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:10.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:15.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:20.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:25.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:30.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:35.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:40.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:45.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:50.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:55.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:00.634 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:05.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:10.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:15.636 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:20.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:25.639 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:30.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:35.656 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:40.636 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:45.636 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:50.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:55.637 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:58:00.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:58:05.636 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:58:10.638 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:58:15.635 -0700 INFO  CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
0 Karma

andrewdwest
Explorer
[clustering]

mode = [master|slave|searchhead|disabled]
* Sets operational mode for this cluster node.
* Only one master may exist per cluster.
* Defaults to disabled.

master_uri = [ | clustermaster:stanzaName1, clustermaster:stanzaName2]
* Only valid for mode=slave or searchhead
* URI of the cluster master that this slave or searchhead should connect to.
* An example of : ://:
* Only for mode=searchhead - If the searchhead is a part of multiple
  clusters, the master uris can be specified by a comma separated list.
0 Karma

andrewdwest
Explorer

https://localhost:8089/servicesNS/admin/search/cluster/master/peers/_new

Splunk Atom Feed: clustermasterpeers
Updated: 2016-08-24T13:56:07-07:00 Splunk build: 6.4.3
Feed links: create - _acl -
_new
active_bundle_id

add_type

base_generation_id

buckets
eai:acl
app search
can_change_perms 1
can_list 1
can_share_app 1
can_share_global 1
can_share_user 1
can_write 1
modifiable 1
owner admin
perms

read

admin
splunk-system-role
write

admin
splunk-system-role
removable 1
sharing user
eai:attributes

optionalFields

forwarderdata_rcv_port
forwarderdata_use_ssl
latest_bundle_id
register_forwarder_address
register_replication_address
register_search_address
replications
site
splunk_version
splunkd_build_number
requiredFields

active_bundle_id
add_type
base_generation_id
buckets
mgmt_port
name
replication_port
replication_use_ssl
server_name
status
wildcardFields

forwarderdata_rcv_port

forwarderdata_use_ssl

latest_bundle_id

mgmt_port

name

register_forwarder_address

register_replication_address

register_search_address
replication_port

replication_use_ssl
replications

server_name
site

splunk_version

splunkd_build_number

status

2016-08-24T13:56:07-07:00 | admin

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...