https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220170#M8236 <P>Seems like the cluster master connection is not happening from Search Head. Did anything changes like host/Ip of SH OR cluster master? Could you verify if communication is allowed from SH to cluster master on port 8089?</P> Mon, 03 Oct 2016 18:15:23 GMT somesoni2 2016-10-03T18:15:23Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220169#M8235 <P>My search head cluster is no longer working after an update from 6.4 to 6.5.0 (I think it was the update!). It seemed to work just fine after the update but then I get in today and it is not working. Here is the log messages:</P> <PRE><CODE>10-03-2016 17:10:37.586 +0000 WARN DistributedPeerManagerHeartbeat - Send failure while pushing PK to search peer = <A href="http://10.0.8.7:8089" target="test_blank">http://10.0.8.7:8089</A> , Connect Timeout 10-03-2016 17:10:37.586 +0000 ERROR DistributedPeerManagerHeartbeat - Status 502 while sending public key to cluster search peer <A href="http://10.0.8.8:8089:" target="test_blank">http://10.0.8.8:8089:</A> 10-03-2016 17:10:37.586 +0000 WARN DistributedPeerManagerHeartbeat - Send failure while pushing PK to search peer = <A href="http://10.0.8.71:8089" target="test_blank">http://10.0.8.71:8089</A> , Connect Timeout 10-03-2016 17:10:37.586 +0000 ERROR DistributedPeerManagerHeartbeat - Status 502 while sending public key to cluster search peer <A href="http://10.0.8.7:8089:" target="test_blank">http://10.0.8.7:8089:</A> </CODE></PRE> <P>Please advise, it seems as though something happened to SSL in the update.</P> Mon, 03 Oct 2016 17:25:43 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220169#M8235 brent_weaver 2016-10-03T17:25:43Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220170#M8236 <P>Seems like the cluster master connection is not happening from Search Head. Did anything changes like host/Ip of SH OR cluster master? Could you verify if communication is allowed from SH to cluster master on port 8089?</P> Mon, 03 Oct 2016 18:15:23 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220170#M8236 somesoni2 2016-10-03T18:15:23Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220171#M8237 <P>I am able to communicate to the master on port 8089... I am wondering is I did the upgrade wrong. I see conflicting info that I needed to break the shcluster before upgrading it... is this the case? If so I did not do that, how do we deal with that situation?</P> Mon, 03 Oct 2016 18:19:05 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220171#M8237 brent_weaver 2016-10-03T18:19:05Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220172#M8238 <P>See if you can manually distribute publick keys to fix this. <BR /> <A href="http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/Configuredistributedsearch#Distribute_the_key_files">http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/Configuredistributedsearch#Distribute_the_key_files</A></P> Mon, 03 Oct 2016 18:26:27 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220172#M8238 somesoni2 2016-10-03T18:26:27Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220173#M8239 <P>Did you upgrade all members of the cluster?<BR /><BR /> Here is the procedure for upgrading from 6.4 to 6.5 <A href="http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/UpgradeaSHC">http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/UpgradeaSHC</A></P> Mon, 03 Oct 2016 18:29:14 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220173#M8239 lukejadamec 2016-10-03T18:29:14Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220174#M8240 <P>Yes I did as per the document. </P> Mon, 03 Oct 2016 18:30:13 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220174#M8240 brent_weaver 2016-10-03T18:30:13Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220175#M8241 <P>the cluster is broken and I am trying to re-add the nodes and it thinks they are already in the cluster, which would make sense. How do I delete "ghost" nodes in the cluster?</P> Mon, 03 Oct 2016 18:35:43 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220175#M8241 brent_weaver 2016-10-03T18:35:43Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220176#M8242 <P>So i think the issue here is with one of the indexing servers... Here is an entry from plunked.log on that server (which is showing as offline in spunk UI under index clustering):</P> <PRE><CODE>10-03-2016 20:07:52.686 +0000 WARN CMSlave - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=logmaster.gehccloud.com:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=A4631FE13867828214C38927C4758A0C add_type=Initial-Add base_generation_id=0 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=1 latest_bundle_id=A4631FE13867828214C38927C4758A0C mgmt_port=8089 name=2FA4A693-FFF5-4D18-87A1-AE33D195C81C register_forwarder_address= register_replication_address= register_search_address= replication_port=8080 replication_use_ssl=0 replications= server_name=hdopeusvmlogi1a site=default splunk_version=6.5.0 splunkd_build_number=59c8927def0f status=Up } ]. </CODE></PRE> <P>It looks like this one index peer node cannot add itself to the index cluster. So this seems to be a problem, if not the problem with the search head?!?!</P> Mon, 03 Oct 2016 20:11:45 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220176#M8242 brent_weaver 2016-10-03T20:11:45Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220177#M8243 <P>This has been resolved. It turns out that a teammate of mine made some network changes and no one was aware. What was actually done, I don't know, but what I do know is that it works now.</P> Tue, 04 Oct 2016 19:20:05 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220177#M8243 brent_weaver 2016-10-04T19:20:05Z https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220178#M8244 <P>any solution for above issue ? i am also getting the same</P> Sun, 08 Apr 2018 21:26:53 GMT https://community.splunk.com/t5/Deployment-Architecture/Why-is-my-search-head-cluster-not-working-after-updating-to/m-p/220178#M8244 splunk24 2018-04-08T21:26:53Z