Deployment Architecture

Best practice to move Splunk indexers servers.

GersonGarcia
Path Finder

Hello all,

We have Splunk Multisite Indexer Cluster in 2 different data centers. Each Site has 3 nodes in the Cluster running Splunk Enterprise 7.3.2.

We are closing down one of the sites and I need to move these 3 indexers to a third site we are moving to.

It would take up to 7 days to have the hosts moved from one site to another, racked and renamed / re-ip.

Our clustering factors are:

site_replication_factor = origin:2,total:3
site_search_factor = origin:2,total:3

What should be the best approach:

1) Move one host at time, wait for data replication is completed and move the next?

2) Move all hosts at the same time and add them back to the cluster one at time?

3) Do I need to place the cluster in Maintenance Mode before move?

4) At the end of move we will keep the 2 Sites environment. Should I create a new Site and move the indexers to there?

Thank you very much,

Gerson Garcia

 

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Here's what I would do.  Others should feel free to offer better suggestions.

0) Move the Cluster Manager off the closing site, if necessary.

1) Put the moving indexers into manual detention.  This will keep them from accepting new data.

2) Shut down the old indexers one at a time using the splunk offline --enforce-counts command.  Wait for each to stop itself before stopping the next.

3) Move the hardware to the site location.

3.5) Consider changing the Site SF/RF to Total:2 to avoid a week of error messages about the RF not being met.

4) Reinstall Splunk on each moved server and re-join it to the cluster.

4.5) Restore the Site SF/RF.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...