https://community.splunk.com/t5/Getting-Data-In/Splunk-Intermitten-Indexer-Cluster-issues-after-upgrade-to-7-2-1/m-p/377811#M68346 <P>Yes, you most likely have one or more pass4SymmKey values set incorrectly.</P> <P>Ensure these values are set correctly within /opt/splunk/etc/system/local/server.conf</P> <P>On the master:</P> <PRE><CODE>[general] serverName = fqdn_of_your_master pass4SymmKey = cluster_wide_password_shared_across_all_nodes [clustering] mode = master replication_factor = int_value search_factor = int_value pass4SymmKey = password_unique_to_this_index_cluster # Note, this is NOT the same p/w as under [general] cluster_label = a_unique_label_for_your_index_cluster # Note, indexers only [indexer_discovery] pass4SymmKey = password_unique_to_indexer_discovery # assuming you use this </CODE></PRE> <P>On each indexer:</P> <PRE><CODE>[general] serverName = fqdn_of_your_indexer pass4SymmKey = cluster_wide_password_shared_across_all_nodes [clustering] mode = slave pass4SymmKey = password_unique_to_this_index_cluster master_uri = full_uri_and_port_of_your_master # e.g. <A href="https://mymasternode.com:8089" target="test_blank">https://mymasternode.com:8089</A> </CODE></PRE> <P>Then cycle the master, and each indexer one by one.</P> Fri, 17 May 2019 19:57:40 GMT codebuilder 2019-05-17T19:57:40Z https://community.splunk.com/t5/Getting-Data-In/Splunk-Intermitten-Indexer-Cluster-issues-after-upgrade-to-7-2-1/m-p/377810#M68345 <P>05-17-2019 00:35:38.768 -0700 WARN CMSlave - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=clustermaster:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=171.67.54.9 mgmtport=8089 (reason: non-zero pending job count=10, guid=39064DB5-4A78-4111-AB9A-DA5A8B7A886B). [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=79DE6CD58B53E34AED36B939B94F8E23 add_type=ReAdd-As-Is base_generation_id=9900 batch_serialno=1 batch_size=154 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=10519 latest_bundle_id=79DE6CD58B53E34AED36B939B94F8E23 mgmt_port=8089 name=39064DB5-4A78-4111-AB9A-DA5A8B7A886B register_forwarder_address= register_replication_address= register_search_address= replication_port=8091 replication_use_ssl=0 replications=client_som_irt_unix~480~B040E664-4DDA-4073-8FD5-4A68619C94C3::39064DB5-4A78-4111-AB9A-DA5A8B7A886B::D9480470-6D8A-4D8B-B6A6-5EF0F926676D server_name=splunkidx04.domain.com site=site1 splunk_version=7.2.1 splunkd_build_number=be11b2c46e23 status=Up } ].</P> <P>Has anyone experienced a similar issue after the upgrading from Splunk Enterprise 6.5.4 to 7.2.1 in a index clustered environment?</P> Wed, 30 Sep 2020 00:35:45 GMT https://community.splunk.com/t5/Getting-Data-In/Splunk-Intermitten-Indexer-Cluster-issues-after-upgrade-to-7-2-1/m-p/377810#M68345 johnward4 2020-09-30T00:35:45Z https://community.splunk.com/t5/Getting-Data-In/Splunk-Intermitten-Indexer-Cluster-issues-after-upgrade-to-7-2-1/m-p/377811#M68346 <P>Yes, you most likely have one or more pass4SymmKey values set incorrectly.</P> <P>Ensure these values are set correctly within /opt/splunk/etc/system/local/server.conf</P> <P>On the master:</P> <PRE><CODE>[general] serverName = fqdn_of_your_master pass4SymmKey = cluster_wide_password_shared_across_all_nodes [clustering] mode = master replication_factor = int_value search_factor = int_value pass4SymmKey = password_unique_to_this_index_cluster # Note, this is NOT the same p/w as under [general] cluster_label = a_unique_label_for_your_index_cluster # Note, indexers only [indexer_discovery] pass4SymmKey = password_unique_to_indexer_discovery # assuming you use this </CODE></PRE> <P>On each indexer:</P> <PRE><CODE>[general] serverName = fqdn_of_your_indexer pass4SymmKey = cluster_wide_password_shared_across_all_nodes [clustering] mode = slave pass4SymmKey = password_unique_to_this_index_cluster master_uri = full_uri_and_port_of_your_master # e.g. <A href="https://mymasternode.com:8089" target="test_blank">https://mymasternode.com:8089</A> </CODE></PRE> <P>Then cycle the master, and each indexer one by one.</P> Fri, 17 May 2019 19:57:40 GMT https://community.splunk.com/t5/Getting-Data-In/Splunk-Intermitten-Indexer-Cluster-issues-after-upgrade-to-7-2-1/m-p/377811#M68346 codebuilder 2019-05-17T19:57:40Z