Migration of Splunk to different server(same platform Linux but with different IP and hostname)

[dvohra]

Hi All,

We are planning to migrate entire Splunk environment to new servers next week and need step by step process. The below document is not quite  helpful to understand the migration. Could anyone please provide us the procedure based on our environment.

https://docs.splunk.com/Documentation/Splunk/8.1.1/Installation/MigrateaSplunkinstance

Architecture(Linux) :-

Server1 - Cluster master and Deployer with different Splunk instance
Server2 - Search head 1 (SHC)
Server3 - Search head 2 (SHC)
Server4 - Search head 3 (SHC)
Server5 - Indexer 1 (Indexer clustering)
Server6 - Indexer 2 (Indexer clustering)

@gcusello @somesoni2 

BR,

Devang

[isoutamo · Accepted solution]

Hi

Here is how we did (multisite)cluster + SHC cluster migration.

1. Ensure that all your configurations have done with DNS names not IPs! If not fix this first.
2. Migrate CM
   1. Set up new CM 
   2. Put cluster into maintenance mode
   3. Stop old CM
   4. Copy old configs to new (etc/system/local, etc/apps/<your own apps>, etc/master-apps, var/run/splunk/cluster/remote-bundle, splunk.secret )
   5. Update DNS CNAME to point to the new IP
   6. Start new
3. Install new IDXc nodes (e.g. https://community.splunk.com/t5/Deployment-Architecture/Swap-indexers-from-indexer-cluster-with-new-...)
   1. Add those to cluster
   2. Rebalance data
   3. Remove old from cluster
4. Install new Deployer
   1. Copy/Restore configurations: https://docs.splunk.com/Documentation/Splunk/7.3.3/DistSearch/PropagateSHCconfigurationchanges#How_t...
5. Install new SHC nodes and add those to SHC
6. Remove old SHC nodes

With those we did it without service breaks for users.

r. Ismo

[dm1]

Can you please elaborate more on the SHC migration part ?

After new deployer is installed and old deployer is migrated to new deployer, are you suggesting to add old SHC nodes to new Deployer, then add new SHC nodes as well to the deployer. once full replication is done on new SHC nodes, remove old SHC nodes ?

[dm1]

hi @isoutamo How can we add new indexer nodes with a different OS version in an existing IDXC with different OS version? E.g. We want to migrate Indexers installed on AWS Linux to Ubuntu 20.04. In this case, how will step 3 work ?

[isoutamo]

As @PickleRick already said, as long as those are in same "OS family" (like linux) and kernel and type is supported then just add / remove etc to/from cluster. But try to keep time with different distro/versions as short as possible.

[PickleRick]

From Splunk's perspective, manipulating peers within the cluster works the same regardless of whether they are on Debian, Centos or gentoo (I wonder if anyone runs Splunk on gentoo ;-)). You should have all nodes on the same distro and release (for example, all nodes on RHEL9) for maintainability purposes but technically - especially during transition period - you can just add Ubuntu peers to AWS Linux-based indexer cluster and they should run just fine.

[WorapongJ]

From Step No.3 Install new Indexer nodes Please correct me if I'm wrong, The overall step that you mention are

1. Add all new Indexers to the same cluster.

2. Increase the replicate data between Indexer.

#CM
[clustering]  
max_peer_build_load = 20 (default 2)
max_peer_rep_load = 50 (default 5)

3. Rebalance the data to reduce the bucket size on the old indexer and make copies of the data to the new indexer.

4. Put one of the old indexers in manual detention to prevent data replication to the old indexer

!!Do this one by one
splunk edit cluster-config -manual_detention on

5. Use the splunk offline --enforce-counts command to stop the indexer and force the Cluster Master to copy the remaining primary buckets to the new indexer.

!!Do this one by one
splunk offline --enforce-counts

6. Remove the old indexer from cluster.

!!Do this one by one
splunk remove cluster-peers -peers <peer_id>

[isoutamo]

Those are ok steps. If you are updating those *_load values, you should remember decrease those when everything is ready.

[WorapongJ]

@isoutamo Thanks you so much, How can I estimate the time required for replicating the data?

[isoutamo]

You could try to calculate transfer time based your network and disk I/O values. Or just start that work and estimate it after sometimes.

[dvohra]

Thank you for providing the detailed procedure. Couple of questions:-

1.  When you migrated Splunk Enterprise to new servers, did you just copy/paste the configs. SHC(OLD) to SHC(NEW), Indexer(OLD) to Indexer(New) etc and then install Splunk over it OR first install Splunk and then copy/paste OR created a new CM,SHC,Indexer just like a new architect and copy the configs.

2.  You mentioned no user was impacted so did you managed to complete the activity same day ?

3.  I believe you have updated the Splunk forwarders to point to the indexers just after the activity.

It seems while you migrated instance one by one you made sure that Splunk is able to communicate with CM(NEW) and SHC/INDEXER(OLD). Is it correct understanding ?

Thanks.

[isoutamo]

1. First install splunk to the new server, then copy needed configurations for CM and Deployer. Indexers and SHC nodes was new installation and then we stretch those clusters by adding new nodes to them and after that removed permanently old nodes. No need to copy anything just migrating data and configurations by splunk cluster features.

2. In our cases that takes couple of weeks as we have hundreds of TBs to migrate from old indexers to the new ones (actually that was migration from one service provider to another).

3. we are using indexer discovery, so that has done automatically.

That was correct understanding.

[dvohra]

@isoutamo I have small confusion on the steps to migrate index clustering(3 servers) to new hardware. I am not able to find any procedure or Splunk docs. Could you please help me out with the steps if possible. Thanks