Why am I getting a warning from my search head cluster captian stating "unable to distribute to peer"?

[w199284]

I'm attempting to convert from a search head (sh) pool to a search head cluster. All instances (cluster master, index peers, heavy forwarders and the original sh pool) are at v6.5.3 on linux. I've followed the steps in the migrate from pool to cluster documentation, carefully I think, a couple of times now. I've missed "something" but I don't know how to find what that is.

I turned on DEBUG for DistributedBundleReplicationManager but didn't find any extra useful information. Same thing for SearchPeerBundlesSetup on one of the peers. To me, it looks like the bundle replication process is working from the sh cluster to the search peer(s) but whatever response is expected from the peer is not happening. Just a wag though. Any thoughts you have on the subject are much appreciated.

o Sending done. uploaded_bytes=82954240, elapsed_ms=5594. Waiting for peer.uri=https://xx.xx.xx.xx:8089 to respond
o got non-200 response from peer. uri=https://xx.xx.xx.xx:8089, reply="HTTP/1.1 204 No Content" response_code=204
o Unable to upload bundle to peer named xxxxx

[esix_splunk]

Did this member successfully join the SHC? If so, you can try to remote it from the cluster, clean, and rejoin it to the cluster.

[w199284]

Thank you for your response. Unless I am missing something, yes all four members of the shc are participating. At least based on the results of shcluster-status.

Actually, I did execute the "clean" command, without options, earlier, on ALL shc members. Very scary command, I think. I had to reinitialize the cluster members afterward to get the members back. (use with caution is right). Since I have not added the shcluster members to the load balancer yet there was no impact.

I still get the bundle failure unfortunately. There are some things that don't add up too like I don't see the Monitoring Console or the shclustering dashboards that should be there. I believe I'll take down the instances and step through the install and configure one more time. Thanks again.