Hi DEAD_BEEF,
just a couple of clues, on where you might be missing some additional time:
- rebuilding the cluster is not high priority. Depending on the indexing and search load, the indexers will nice the bucket replications
- The data is not being transferred in bulk, but bucket per bucket. Every bucket received at the rebuilt target will be reprocessed locally before the next bucket is being fetched. Expect some processing overhead on both sides
- Depending if the rebuilt indexes will be searchable, the target has to rebuild the tsidx files that easily add up to 33% of the original bucket data
- The most current buckets need to roll in order to be replicated
- Depending on the speed of the Warm/Cold drives (where most of the data goes) one of your bottlenecks on both the sending and the receiving indexers will be the disks. Check the IO throughput on the warm/cold storage of your production indexers. I would bet that they can not saturate a 10Gbps link
In summary, given the question at hand, I suggest that the sustained I/O bandwidth of the warm/cold storage will determine the overall time to restore the indexer and should give you a rough estimate. If you have done a restore before, divide the data amount by the bandwidth of the drives. The difference to the observed time will be the processing overhead due to the factors above. With the overhead in % of the total time estimated for the data/bandwidth and the current data volume, you should be good to go.
Oliver
... View more