Bucket replication queue full causing random indexer slowdown.

sbhale — Mon, 04 Feb 2019 20:16:07 GMT

Had a weird issue where my queues would fill up on random nodes and rove around within the cluster.
Had a case opened with support and Was working through and making all sorts of adjustments and ruling out all sorts of issues to no vail.
Finally had a breakthrough when I noticed that we were seeing
INFO BucketReplicator - replication queue for peer=<guid> bid=<bid> is full.
Followed almost immediately by
INFO BucketReplicator - replication queue for peer=<guid> bid=<bid> has room now.
over and over again. The gap between those two messages was only a few milliseconds.

No other obvious ERROR pointing to the cause.

Re: Bucket replication queue full causing random indexer slowdown.

sbhale — Mon, 04 Feb 2019 20:17:58 GMT

Answering my own question so others will find it useful.

The presense of the above messages with the same peer guid was ruled to be the problem.
One of our peer nodes was acting up and slowing down any nodes replicating to it just a little bit but enough that it was a propagating and causing queues to get backed up.
The solution was putting the node in manual detention to be either re-built or retired.

topic Bucket replication queue full causing random indexer slowdown. in Getting Data In

Bucket replication queue full causing random indexer slowdown.

Re: Bucket replication queue full causing random indexer slowdown.