topic Cluster bucket rebalancing: How long is too long? in Deployment Architecture

Cluster bucket rebalancing: How long is too long?

twinspop — Tue, 04 Apr 2017 14:06:11 GMT

My first few attempts at rebalancing were pretty great. No muss, no fuss. They ran for about 12 hours and like magic my cluster was firing on all cylinders. Beautiful.

'Stuff happens' and I'm now in the situation again where I've introduced new servers to the the cluster (replacing old). Now I'm way out of balance. "No problem," says I. "Data rebalancing is awesome."

Not so fast. Literally. I fired it off late Friday night. By Monday morning the process was reading 0.14% done, .01% more than right after starting the process 56 hours previously. By my math that's about 650 days to complete.

I stopped the process, and restarted it for 1 index only -- 648 buckets using 1 TB of disk. After running for the last 18 hours, it's at 3% complete. So slow as to not really be usable.

12 servers in the cluster, 4 are new; all are:

20 core Xeons (2 threads each)
22 1.6TB SSDs
128 GB of RAM
Splunk 6.5.1
Avg load over the weekend was under 2

Any suggestions appreciated.

EDIT:

> splunk btool server list clustering | grep max_peer
max_peer_build_load = 2
max_peer_rep_load = 5
max_peer_sum_rep_load = 5

Jon

Re: Cluster bucket rebalancing: How long is too long?

somesoni2 — Tue, 29 Sep 2020 13:32:00 GMT

Try to use the lower rebalance_threshold on the master to see if the rebalance performance improves. You can rebalance in multiple waves by increasing the rebalance_threshold.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Rebalancethecluster#Configure_the_data_rebalancing_threshold

Re: Cluster bucket rebalancing: How long is too long?

twinspop — Tue, 04 Apr 2017 16:13:43 GMT

Attempting at 0.5. I'll get back with results.

Re: Cluster bucket rebalancing: How long is too long?

mattlucas719 — Thu, 06 Apr 2017 15:46:18 GMT

What is your retention policy and your replication factor?

Re: Cluster bucket rebalancing: How long is too long?

sk314 — Mon, 26 Jun 2017 23:22:55 GMT

Hah! @twinspop I seem to be following you from https://answers.splunk.com/answers/476015/whats-the-best-method-to-updatereplace-indexer-clu.html

I am in the same spot now and indexer rebalance is painfully slow. Did you find out the cause for this?

Re: Cluster bucket rebalancing: How long is too long?

twinspop — Tue, 27 Jun 2017 01:35:32 GMT

My cold storage was on NAS. Since having all stages (hot, warm, cold) on local drives, rebalance is plenty fast.

Re: Cluster bucket rebalancing: How long is too long?

twinspop — Tue, 27 Jun 2017 01:36:28 GMT

The problem (apparently) was related to COLD storage being on NAS. Since restructuring our storage plans to get everything on local drives, data rebalance is a quick process.

Re: Cluster bucket rebalancing: How long is too long?

sk314 — Tue, 27 Jun 2017 01:39:36 GMT

Hmmm - we have all our storage on local drives - except frozen/archive - which is on NFS. Does frozen/archive drive count?

Re: Cluster bucket rebalancing: How long is too long?

twinspop — Tue, 27 Jun 2017 01:42:54 GMT

No, frozen data isn't involved in data rebalance.