Splunk Answers,
I started receiving this message on August 23rd, 2016. I am not sure why the peers will not join cluster?
Waiting for requisite number of peers to join the cluster - https://127.0.0.1:8089. Cluster has only 0 peers (waiting for 3 peers to join the cluster)
Please advise how to resolve the issue, thank you in advance.
The message/problem was being caused by a setting under "Settings" -> "Distributed Environment" -> "Distributed Search" -> "Distributed search set up" -> Change the setting from "Yes" to "No."
Andrewwest,
Is this on the CM or indexers? I have the same issue, but I am not able to login to the CM via GUI so I am having a hard time figuring it out. Let me know where to change it.
The message/problem was being caused by a setting under "Settings" -> "Distributed Environment" -> "Distributed Search" -> "Distributed search set up" -> Change the setting from "Yes" to "No."
It's not necessary to disable to fix the issue. The issue can be related to the communication between the Indexer and Cluster Master. It can also occur when the Splunk service isn't running on the Indexers.
so the indexer connects for about 10 sec then goes down. What are some good troubleshoot steps to help me resolve this?
First step is to check the logs on the Cluster Master as well as on the Indexers.
Second step is apply the cluster bundle from Cluster Master to the Indexers to ensure the configuration regarding the apps is synchronous among the peers.
Third step, whenever the indexer drops the connection check the splunk service status.
08-24-2016 14:31:59.193 -0700 INFO CMMaster - event=service status=skipping reason='Cluster has only 0 peers (waiting for 1 peers to join the cluster)'
08-24-2016 14:52:50.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:52:55.636 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:00.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:05.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:10.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:15.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:20.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:25.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:30.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:30.785 -0700 INFO CMMaster - event=service status=skipping reason='Cluster has only 0 peers (waiting for 1 peers to join the cluster)'
08-24-2016 14:53:31.852 -0700 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/Applications/Splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
08-24-2016 14:53:31.861 -0700 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/Applications/Splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
08-24-2016 14:53:35.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:40.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:45.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:50.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:53:55.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:00.640 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:05.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:10.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:15.640 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:20.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:25.640 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:30.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:35.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:40.636 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:45.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:50.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:54:55.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:00.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:05.636 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:10.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:15.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:20.640 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:25.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:30.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:35.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:40.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:45.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:50.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:55:55.634 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:00.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:05.634 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:10.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:15.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:20.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:25.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:30.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:35.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:40.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:45.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:50.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:56:55.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:00.634 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:05.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:10.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:15.636 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:20.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:25.639 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:30.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:35.656 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:40.636 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:45.636 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:50.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:57:55.637 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:58:00.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:58:05.636 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:58:10.638 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
08-24-2016 14:58:15.635 -0700 INFO CMConfig - buckets_to_sum=primaries buckets_to_sum_uint=primaries
[clustering]
mode = [master|slave|searchhead|disabled]
* Sets operational mode for this cluster node.
* Only one master may exist per cluster.
* Defaults to disabled.
master_uri = [ | clustermaster:stanzaName1, clustermaster:stanzaName2]
* Only valid for mode=slave or searchhead
* URI of the cluster master that this slave or searchhead should connect to.
* An example of : ://:
* Only for mode=searchhead - If the searchhead is a part of multiple
clusters, the master uris can be specified by a comma separated list.
https://localhost:8089/servicesNS/admin/search/cluster/master/peers/_new
Splunk Atom Feed: clustermasterpeers
Updated: 2016-08-24T13:56:07-07:00 Splunk build: 6.4.3
Feed links: create - _acl -
_new
active_bundle_id
add_type
base_generation_id
buckets
eai:acl
app search
can_change_perms 1
can_list 1
can_share_app 1
can_share_global 1
can_share_user 1
can_write 1
modifiable 1
owner admin
perms
read
admin
splunk-system-role
write
admin
splunk-system-role
removable 1
sharing user
eai:attributes
optionalFields
forwarderdata_rcv_port
forwarderdata_use_ssl
latest_bundle_id
register_forwarder_address
register_replication_address
register_search_address
replications
site
splunk_version
splunkd_build_number
requiredFields
active_bundle_id
add_type
base_generation_id
buckets
mgmt_port
name
replication_port
replication_use_ssl
server_name
status
wildcardFields
forwarderdata_rcv_port
forwarderdata_use_ssl
latest_bundle_id
mgmt_port
name
register_forwarder_address
register_replication_address
register_search_address
replication_port
replication_use_ssl
replications
server_name
site
splunk_version
splunkd_build_number
status
2016-08-24T13:56:07-07:00 | admin