Deployment Architecture

Best practice to move Splunk indexers servers.

GersonGarcia
Path Finder

Hello all,

We have Splunk Multisite Indexer Cluster in 2 different data centers. Each Site has 3 nodes in the Cluster running Splunk Enterprise 7.3.2.

We are closing down one of the sites and I need to move these 3 indexers to a third site we are moving to.

It would take up to 7 days to have the hosts moved from one site to another, racked and renamed / re-ip.

Our clustering factors are:

site_replication_factor = origin:2,total:3
site_search_factor = origin:2,total:3

What should be the best approach:

1) Move one host at time, wait for data replication is completed and move the next?

2) Move all hosts at the same time and add them back to the cluster one at time?

3) Do I need to place the cluster in Maintenance Mode before move?

4) At the end of move we will keep the 2 Sites environment. Should I create a new Site and move the indexers to there?

Thank you very much,

Gerson Garcia

 

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Here's what I would do.  Others should feel free to offer better suggestions.

0) Move the Cluster Manager off the closing site, if necessary.

1) Put the moving indexers into manual detention.  This will keep them from accepting new data.

2) Shut down the old indexers one at a time using the splunk offline --enforce-counts command.  Wait for each to stop itself before stopping the next.

3) Move the hardware to the site location.

3.5) Consider changing the Site SF/RF to Total:2 to avoid a week of error messages about the RF not being met.

4) Reinstall Splunk on each moved server and re-join it to the cluster.

4.5) Restore the Site SF/RF.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...