Deployment Architecture

Best practice to move Splunk indexers servers.

Path Finder

Hello all,

We have Splunk Multisite Indexer Cluster in 2 different data centers. Each Site has 3 nodes in the Cluster running Splunk Enterprise 7.3.2.

We are closing down one of the sites and I need to move these 3 indexers to a third site we are moving to.

It would take up to 7 days to have the hosts moved from one site to another, racked and renamed / re-ip.

Our clustering factors are:

site_replication_factor = origin:2,total:3
site_search_factor = origin:2,total:3

What should be the best approach:

1) Move one host at time, wait for data replication is completed and move the next?

2) Move all hosts at the same time and add them back to the cluster one at time?

3) Do I need to place the cluster in Maintenance Mode before move?

4) At the end of move we will keep the 2 Sites environment. Should I create a new Site and move the indexers to there?

Thank you very much,

Gerson Garcia


Labels (1)
0 Karma


Here's what I would do.  Others should feel free to offer better suggestions.

0) Move the Cluster Manager off the closing site, if necessary.

1) Put the moving indexers into manual detention.  This will keep them from accepting new data.

2) Shut down the old indexers one at a time using the splunk offline --enforce-counts command.  Wait for each to stop itself before stopping the next.

3) Move the hardware to the site location.

3.5) Consider changing the Site SF/RF to Total:2 to avoid a week of error messages about the RF not being met.

4) Reinstall Splunk on each moved server and re-join it to the cluster.

4.5) Restore the Site SF/RF.

If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...