Splunk Enterprise

Migration of Splunk to different server(same platform Linux but with different IP and hostname)

dvohra
Explorer

Hi All,

We are planning to migrate entire Splunk environment to new servers next week and need step by step process. The below document is not quite  helpful to understand the migration. Could anyone please provide us the procedure based on our environment.

https://docs.splunk.com/Documentation/Splunk/8.1.1/Installation/MigrateaSplunkinstance

Architecture(Linux) :-

Server1 - Cluster master and Deployer with different Splunk instance
Server2 - Search head 1 (SHC)
Server3 - Search head 2 (SHC)
Server4 - Search head 3 (SHC)
Server5 - Indexer 1 (Indexer clustering)
Server6 - Indexer 2 (Indexer clustering)

@gcusello @somesoni2 

BR,

Devang

0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

Here is how we did (multisite)cluster + SHC cluster migration.

  1. Ensure that all your configurations have done with DNS names not IPs! If not fix this first.
  2. Migrate CM
    1. Set up new CM 
    2. Put cluster into maintenance mode
    3. Stop old CM
    4. Copy old configs to new (etc/system/local, etc/apps/<your own apps>, etc/master-apps, var/run/splunk/cluster/remote-bundle, splunk.secret )
    5. Update DNS CNAME to point to the new IP
    6. Start new
  3. Install new IDXc nodes (e.g. https://community.splunk.com/t5/Deployment-Architecture/Swap-indexers-from-indexer-cluster-with-new-...)
    1. Add those to cluster
    2. Rebalance data
    3. Remove old from cluster
  4. Install new Deployer
    1. Copy/Restore configurations: https://docs.splunk.com/Documentation/Splunk/7.3.3/DistSearch/PropagateSHCconfigurationchanges#How_t...
  5. Install new SHC nodes and add those to SHC
  6. Remove old SHC nodes

With those we did it without service breaks for users.

r. Ismo

View solution in original post

isoutamo
SplunkTrust
SplunkTrust

Hi

Here is how we did (multisite)cluster + SHC cluster migration.

  1. Ensure that all your configurations have done with DNS names not IPs! If not fix this first.
  2. Migrate CM
    1. Set up new CM 
    2. Put cluster into maintenance mode
    3. Stop old CM
    4. Copy old configs to new (etc/system/local, etc/apps/<your own apps>, etc/master-apps, var/run/splunk/cluster/remote-bundle, splunk.secret )
    5. Update DNS CNAME to point to the new IP
    6. Start new
  3. Install new IDXc nodes (e.g. https://community.splunk.com/t5/Deployment-Architecture/Swap-indexers-from-indexer-cluster-with-new-...)
    1. Add those to cluster
    2. Rebalance data
    3. Remove old from cluster
  4. Install new Deployer
    1. Copy/Restore configurations: https://docs.splunk.com/Documentation/Splunk/7.3.3/DistSearch/PropagateSHCconfigurationchanges#How_t...
  5. Install new SHC nodes and add those to SHC
  6. Remove old SHC nodes

With those we did it without service breaks for users.

r. Ismo

dvohra
Explorer

Thank you for providing the detailed procedure. Couple of questions:-

1.  When you migrated Splunk Enterprise to new servers, did you just copy/paste the configs. SHC(OLD) to SHC(NEW), Indexer(OLD) to Indexer(New) etc and then install Splunk over it OR first install Splunk and then copy/paste OR created a new CM,SHC,Indexer just like a new architect and copy the configs.

2.  You mentioned no user was impacted so did you managed to complete the activity same day ?

3.  I believe you have updated the Splunk forwarders to point to the indexers just after the activity.

It seems while you migrated instance one by one you made sure that Splunk is able to communicate with CM(NEW) and SHC/INDEXER(OLD). Is it correct understanding ?

Thanks.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

1. First install splunk to the new server, then copy needed configurations for CM and Deployer. Indexers and SHC nodes was new installation and then we stretch those clusters by adding new nodes to them and after that removed permanently old nodes. No need to copy anything just migrating data and configurations by splunk cluster features.

2. In our cases that takes couple of weeks as we have hundreds of TBs to migrate from old indexers to the new ones (actually that was migration from one service provider to another).

3. we are using indexer discovery, so that has done automatically.

That was correct understanding.

0 Karma

dvohra
Explorer

@isoutamo I have small confusion on the steps to migrate index clustering(3 servers) to new hardware. I am not able to find any procedure or Splunk docs. Could you please help me out with the steps if possible. Thanks 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...