We have an existing multi-site indexer cluster of 8 indexers, we have got 16 new indexers and have to migrate the current multi-site indexer cluster to the new hardware, what's the best possible way to achieve this indexer data migration?
The intention is to have existing hardware consisting of the 8 indexers (multi-site cluster) decommissioned and have the new indexers replace them without losing any data.
What replication factor do you have in your current cluster?
I guess (but I'm not the expert on this) you could add the new indexers to the existing cluster and then one by one take the old indexers offline and let replication take care of retaining the data (so wait until the replication factor is met again, before decommissioning the next indexer).
More information on taking indexer peers offline: http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Takeapeeroffline
As soon as you have added the new indexers to the cluster, I guess you could put the old indexers in detention (in preparation for taking them offline), so they will no longer take part in indexing incoming data and will no longer replicate buckets from other indexers: http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Peerdetention
Replication factor is 3.
I forgot to mention this in my Question:: is it possible to localize the data replication, between the DCs, we don't wan't TBs of data move between the DCs for the replication?