Search peer ###############.com has the following message: Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=##############.com:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=10.0.0.1 mgmtport=8089 (reason: http client error=Read Timeout, while trying to reach https://10.0.0.1:8089/services/cluster/config). [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=##########################3 add_type=Clear-Masks-And-ReAdd base_generation_id=15418 batch_serialno=1 batch_size=1 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=1 last_complete_generation_id=0 latest_bundle_id=################# mgmt_port=8089 name=##################### register_forwarder_address= register_replication_address= register_search_address= replication_port=9100 replication_use_ssl=0 replications= server_name=#######.com site=default splunk_version=7.2.6 splunkd_build_number=c0bf0f679ce9 status=Up } ].
I tried solving the issue by changing the PassSymm4Key under idx cluster, but still facing same error.
In indexer cluster environment, the search head have to be connected to the cluster master and the pass4SymmKey have to be the same as you had deployed to cluster master and indexers as well.
When I experienced this issue in past installations, I redeploy the cluster configuration again just to make sure there is no mistyped on the pass4Symmkey password. Please remember to restart the splunk service when you are removing the configuration to cleanup any issues. I mean, remove the configuration from cluster master, restart splunk service, remove the configuration from Search Head and restart the splunk service. after that, redeploy the configuration again.