Best practice to move Splunk indexers servers.

[GersonGarcia]

Hello all,

We have Splunk Multisite Indexer Cluster in 2 different data centers. Each Site has 3 nodes in the Cluster running Splunk Enterprise 7.3.2.

We are closing down one of the sites and I need to move these 3 indexers to a third site we are moving to.

It would take up to 7 days to have the hosts moved from one site to another, racked and renamed / re-ip.

Our clustering factors are:

site_replication_factor = origin:2,total:3
site_search_factor = origin:2,total:3

What should be the best approach:

1) Move one host at time, wait for data replication is completed and move the next?

2) Move all hosts at the same time and add them back to the cluster one at time?

3) Do I need to place the cluster in Maintenance Mode before move?

4) At the end of move we will keep the 2 Sites environment. Should I create a new Site and move the indexers to there?

Thank you very much,

Gerson Garcia

[richgalloway]

Here's what I would do.  Others should feel free to offer better suggestions.

0) Move the Cluster Manager off the closing site, if necessary.

1) Put the moving indexers into manual detention.  This will keep them from accepting new data.

2) Shut down the old indexers one at a time using the splunk offline --enforce-counts command.  Wait for each to stop itself before stopping the next.

3) Move the hardware to the site location.

3.5) Consider changing the Site SF/RF to Total:2 to avoid a week of error messages about the RF not being met.

4) Reinstall Splunk on each moved server and re-join it to the cluster.

4.5) Restore the Site SF/RF.