Migration of Splunk to different server(same platform Linux but with different IP and hostname)

[dvohra]

Hi All,

We are planning to migrate entire Splunk environment to new servers next week and need step by step process. The below document is not quite  helpful to understand the migration. Could anyone please provide us the procedure based on our environment.

https://docs.splunk.com/Documentation/Splunk/8.1.1/Installation/MigrateaSplunkinstance

Architecture(Linux) :-

Server1 - Cluster master and Deployer with different Splunk instance
Server2 - Search head 1 (SHC)
Server3 - Search head 2 (SHC)
Server4 - Search head 3 (SHC)
Server5 - Indexer 1 (Indexer clustering)
Server6 - Indexer 2 (Indexer clustering)

@gcusello @somesoni2 

BR,

Devang

[isoutamo · Accepted solution]

Hi

Here is how we did (multisite)cluster + SHC cluster migration.

1. Ensure that all your configurations have done with DNS names not IPs! If not fix this first.
2. Migrate CM
   1. Set up new CM 
   2. Put cluster into maintenance mode
   3. Stop old CM
   4. Copy old configs to new (etc/system/local, etc/apps/<your own apps>, etc/master-apps, var/run/splunk/cluster/remote-bundle, splunk.secret )
   5. Update DNS CNAME to point to the new IP
   6. Start new
3. Install new IDXc nodes (e.g. https://community.splunk.com/t5/Deployment-Architecture/Swap-indexers-from-indexer-cluster-with-new-...)
   1. Add those to cluster
   2. Rebalance data
   3. Remove old from cluster
4. Install new Deployer
   1. Copy/Restore configurations: https://docs.splunk.com/Documentation/Splunk/7.3.3/DistSearch/PropagateSHCconfigurationchanges#How_t...
5. Install new SHC nodes and add those to SHC
6. Remove old SHC nodes

With those we did it without service breaks for users.

r. Ismo

[WorapongJ]

From Step No.3 Install new Indexer nodes Please correct me if I'm wrong, The overall step that you mention are

1. Add all new Indexers to the same cluster.

2. Increase the replicate data between Indexer.

#CM
[clustering]  
max_peer_build_load = 20 (default 2)
max_peer_rep_load = 50 (default 5)

3. Rebalance the data to reduce the bucket size on the old indexer and make copies of the data to the new indexer.

4. Put one of the old indexers in manual detention to prevent data replication to the old indexer

!!Do this one by one
splunk edit cluster-config -manual_detention on

5. Use the splunk offline --enforce-counts command to stop the indexer and force the Cluster Master to copy the remaining primary buckets to the new indexer.

!!Do this one by one
splunk offline --enforce-counts

6. Remove the old indexer from cluster.

!!Do this one by one
splunk remove cluster-peers -peers <peer_id>

[isoutamo]

Those are ok steps. If you are updating those *_load values, you should remember decrease those when everything is ready.

[WorapongJ]

@isoutamo Thanks you so much, How can I estimate the time required for replicating the data?

[isoutamo]

You could try to calculate transfer time based your network and disk I/O values. Or just start that work and estimate it after sometimes.

[dvohra]

Thank you for providing the detailed procedure. Couple of questions:-

1.  When you migrated Splunk Enterprise to new servers, did you just copy/paste the configs. SHC(OLD) to SHC(NEW), Indexer(OLD) to Indexer(New) etc and then install Splunk over it OR first install Splunk and then copy/paste OR created a new CM,SHC,Indexer just like a new architect and copy the configs.

2.  You mentioned no user was impacted so did you managed to complete the activity same day ?

3.  I believe you have updated the Splunk forwarders to point to the indexers just after the activity.

It seems while you migrated instance one by one you made sure that Splunk is able to communicate with CM(NEW) and SHC/INDEXER(OLD). Is it correct understanding ?

Thanks.

[isoutamo]

1. First install splunk to the new server, then copy needed configurations for CM and Deployer. Indexers and SHC nodes was new installation and then we stretch those clusters by adding new nodes to them and after that removed permanently old nodes. No need to copy anything just migrating data and configurations by splunk cluster features.

2. In our cases that takes couple of weeks as we have hundreds of TBs to migrate from old indexers to the new ones (actually that was migration from one service provider to another).

3. we are using indexer discovery, so that has done automatically.

That was correct understanding.

[dvohra]

@isoutamo I have small confusion on the steps to migrate index clustering(3 servers) to new hardware. I am not able to find any procedure or Splunk docs. Could you please help me out with the steps if possible. Thanks