How to migrate clustered indexed data from old server to new server?

[koyachi]

Hi Folks,

We have a Splunk instance which comprises of 1 SH, 2 Indexers in cluster (Replication enabled), 1 Cluster Master and 1 Heavy forwarder.  

We need to migrate these servers to a new set of servers. I would like to know the steps involved in migrating the already indexed clustered data.

I am little newbie in Splunk Administration so would appreciate if someone can help me with detailed steps and points that i need to take care of while migrating the data.

Thanks in advance.

[woodcock]

There are many ways to do this.  The easiest way is to recycle the hot/code volumes and reattach them as-is to the new HW.  Cake.

[koyachi]

This entire instance is setup on Azure and i am not sure if we can recycle disk and attach it to new VMs.

[richgalloway]

The easy way to move an indexer cluster to new hardware is to add new indexers on the new hardware to the cluster and then remove the old indexers from the cluster.  Specifically,

1) Install Splunk on the new hardware and configure it to match the old indexers
2) Add the new indexers to the cluster
3) Put the old indexers into Detention
4) Issue a 'splunk offline --enforce-counts' command to ONE old indexer
5) Wait for the buckets to migrate off the old indexer. Depending on the number of buckets, this could take a while.
6) Repeat steps 4-5 for the remaining old indexers.
7) Once all buckets are moved to the new indexers you can remove the old indexers from the cluster and retire the old hardware.

Moving search heads and management instances is simpler.  Install Splunk on the new hardware and then copy $SPLUNK_HOME/etc from the old instance to the new one.

The final step is to clean up the loose ends, such as pointing everyone to the new CM.  Switching to the new indexers is a non-problem if you're using Indexer Discovery; otherwise, you'll have to change the CM, SHs, and HF to use the new indexer names/addresses.  Don't forget to point everyone to the new License Manager.  If you use DNS then this redirection is trivial.

[PickleRick]

Yep. That's the most "officially correct way to do it. The most challenging part may prove to be reconfiguring your forwarders to point to the new infrastructure. Depending on your setup it might be as trivial as updating a single app distributed to all forwarders or changing DNS records or as complicated as manually reconfiguring each forwarder's outputs.conf.