Splunk Intermitten Indexer Cluster issues after upgrade to 7.2.1

[johnward4]

05-17-2019 00:35:38.768 -0700 WARN CMSlave - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=clustermaster:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=171.67.54.9 mgmtport=8089 (reason: non-zero pending job count=10, guid=39064DB5-4A78-4111-AB9A-DA5A8B7A886B). [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=79DE6CD58B53E34AED36B939B94F8E23 add_type=ReAdd-As-Is base_generation_id=9900 batch_serialno=1 batch_size=154 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=10519 latest_bundle_id=79DE6CD58B53E34AED36B939B94F8E23 mgmt_port=8089 name=39064DB5-4A78-4111-AB9A-DA5A8B7A886B register_forwarder_address= register_replication_address= register_search_address= replication_port=8091 replication_use_ssl=0 replications=client_som_irt_unix~480~B040E664-4DDA-4073-8FD5-4A68619C94C3::39064DB5-4A78-4111-AB9A-DA5A8B7A886B::D9480470-6D8A-4D8B-B6A6-5EF0F926676D server_name=splunkidx04.domain.com site=site1 splunk_version=7.2.1 splunkd_build_number=be11b2c46e23 status=Up } ].

Has anyone experienced a similar issue after the upgrading from Splunk Enterprise 6.5.4 to 7.2.1 in a index clustered environment?

[codebuilder]

Yes, you most likely have one or more pass4SymmKey values set incorrectly.

Ensure these values are set correctly within /opt/splunk/etc/system/local/server.conf

On the master:

[general]
serverName = fqdn_of_your_master
pass4SymmKey = cluster_wide_password_shared_across_all_nodes

[clustering]
mode = master
replication_factor = int_value
search_factor = int_value
pass4SymmKey = password_unique_to_this_index_cluster # Note, this is NOT the same p/w as under [general]
cluster_label = a_unique_label_for_your_index_cluster # Note, indexers only

[indexer_discovery]
pass4SymmKey = password_unique_to_indexer_discovery # assuming you use this

On each indexer:

[general]
serverName = fqdn_of_your_indexer
pass4SymmKey = cluster_wide_password_shared_across_all_nodes

[clustering]
mode = slave
pass4SymmKey = password_unique_to_this_index_cluster 
master_uri = full_uri_and_port_of_your_master # e.g. https://mymasternode.com:8089

Then cycle the master, and each indexer one by one.