Deployment Architecture

Best practice to move Splunk indexers servers.

GersonGarcia
Path Finder

Hello all,

We have Splunk Multisite Indexer Cluster in 2 different data centers. Each Site has 3 nodes in the Cluster running Splunk Enterprise 7.3.2.

We are closing down one of the sites and I need to move these 3 indexers to a third site we are moving to.

It would take up to 7 days to have the hosts moved from one site to another, racked and renamed / re-ip.

Our clustering factors are:

site_replication_factor = origin:2,total:3
site_search_factor = origin:2,total:3

What should be the best approach:

1) Move one host at time, wait for data replication is completed and move the next?

2) Move all hosts at the same time and add them back to the cluster one at time?

3) Do I need to place the cluster in Maintenance Mode before move?

4) At the end of move we will keep the 2 Sites environment. Should I create a new Site and move the indexers to there?

Thank you very much,

Gerson Garcia

 

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Here's what I would do.  Others should feel free to offer better suggestions.

0) Move the Cluster Manager off the closing site, if necessary.

1) Put the moving indexers into manual detention.  This will keep them from accepting new data.

2) Shut down the old indexers one at a time using the splunk offline --enforce-counts command.  Wait for each to stop itself before stopping the next.

3) Move the hardware to the site location.

3.5) Consider changing the Site SF/RF to Total:2 to avoid a week of error messages about the RF not being met.

4) Reinstall Splunk on each moved server and re-join it to the cluster.

4.5) Restore the Site SF/RF.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...