Migrate cluster master to a new server

[josephp]

Hi all,

I would like to migrate our current cluster master to the a new server. Here's what I gather the process to do so. If someone can take a look and let me know if there's anything missing that'll be much appreciated.

Thank you!

Additionally, should I enable cluster maintenance mode on the old cluster master prior to the migration? 

========================================================================
======================== Migrate the Cluster Master ====================
========================================================================
- Stop the splunk service on both the old and new cluster master
/opt/splunk/bin/splunk stop

- On the old Cluster Master change encrypted passwords to clear text and save theses

find /opt/splunk/etc -name '*.conf' -exec grep -inH '\$[0-9]\$' {} \;
/opt/splunk/bin/splunk show-decrypted --value '$encryptedpassword'

- - Copy files to the new Cluster Master
scp -r /opt/splunk/var/run/splunk/cluster/remote-bundle/ new_splunkmaster:/opt/splunk/var/run/splunk/cluster/remote-bundle/
scp -r /opt/splunk/etc/master-apps/ new_splunkmaster:/opt/splunk/etc/
scp -r /opt/splunk/etc/system/local/server.conf new_splunkmaster:/opt/splunk/etc/system/local/

- Make sure the above decrypted the main 2 passwords below and replace them in the copied server.conf, in clear text, on the new Cluster Master until it is restarted when it will then encrypt.

[general]
sslPassword=

[clustering]
pass4SymmKey=

- Start splunk on the new Cluster Master
/opt/splunk/bin/splunk start

- Point indexers to the new Cluster Master
/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://new_splunkmaster:8089 -replication_port 9887 -secret new_splunkmaster

- Point the search heads to the new Cluster Master
/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://new_splunkmaster:8089 -secret new_splunkmaster

========================================================================
======================== Migrate the License Manager ====================
========================================================================

- Promote a license peer to be the manager:

On the peer, navigate to Settings > Licensing.
Click Switch to local manager.
On the Change manager association page, choose Designate this Splunk instance as the manager license server.
Click Save.
Restart the Splunk Enterprise services.
On the new license manager, install your licenses. See Install a license.
Configure the license peers to use the new license manager:

- On the peer (indexer / search heads / deployer), navigate to Settings > Licensing.
Click Switch to local manager.
Update the Manager license server URI to point at the new license manager.
Click Save.
Restart the Splunk Enterprise services.
Demote the old license manager to be a peer:

- On the old license manager, navigate to Settings > Licensing.
Click Change to peer.
Click Designate a different Splunk instance as the manager license server.
Update the Manager license server URI to point at the new license manager.
Click Save.
Stop the Splunk Enterprise services.
Using the CLI, delete any license files under $SPLUNK_HOME/etc/licenses/enterprise/.
Start the Splunk Enterprise services.

[isoutamo]

Hi

here is how I did it. I actually migrate the whole distributed multisite environment from one service provider to another. https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platf...
r. Ismo

[josephp]

Thanks. It looks like you migrate the IP but keep the DNS names. We'll be moving both. If we issue maintenance mode on the old Cluster Manager, then migrate, how would we ensure the maintenance mode is lifted after moving to the new one? 

[isoutamo]

Maintenance mode is one parameter in server.conf file. So when you copy it to target it will be there. Then just disable maintenance mode and it will removed from server.conf.

If you change both name and ip there could be issues as all peers and SHs are using name or ip to recognize the cluster! I’m not 100% sure if the peers is actually recognized by GUID, but I almost proposed you to do offline not online migration and you need change this to all peers before start them. Also same for other components/nodes.

[PickleRick]

1. Why would you fiddle with license manager? (unless it's on the CM which is not a very good idea)

2. Why copy anything from var/run?

3. Switching indexers between CMs is asking for trouble. I'd replace a CM in place.

[isoutamo]

Var/run contains some status information. Everything works smoothlier with it, but it’s not a catastrophe without it.

[josephp]

Hi the cluster master is also our License manager. 

And by replacing a CM in place, you mean keeping the IPs and DNS of the CM?

Copy from /var/run is listed in the https://docs.splunk.com/Documentation/Splunk/9.4.0/Indexer/Handlemanagernodefailure