I am looking for a least disruptive process to get back the data.
Can this problem be solved using a standalone search peer which is just for this purpose? In some places I've worked putting the production cluster into maintenance mode to load old data is simply not an option.
There must be a better way. It seems needlessly restrictive and complicated to have to restore the buckets on the cluster nodes it first came from. Enhancement needed?
please read here:
as for question 2, never restart a single indexer in the cluster by itself. you can go for maintenance mode, thaw the buckets to the indexer they were rolled out from - make sure GUID matches, restart that indexer and then disable maintenance mode.
read more here:
as for question 3, the challenge on test machine is that the buckets are marked with the indexers guid
would recommend to do the following:
find all data needed to be thawed, best would be if its all from same indexer.
place cluster in maintenance mode, thaw the data, get verify its searchable, disable maintenance mode
hope it helps