Migrate indexer database to new cluster

[Darthsplunker]

Hello, all!

Im hopefully looking for an ELI5 (explain like im 5) on the best way to migrate indexer cluster database to an entirely new cluster environment. The end goal is to decommission the current setup.

My current setup.  RHEL 7, physical,  splunk 8.2.4. All log sources are still flowing to this setup.
3sh cluster, 3 idx cluster, 1cm, etc.

New: RHEL 8,  AWS/VM's, splunk 9.1.1. This setup is still empty with no logs/sources flowing here yet.
3sh cluster, 3 idx cluster, 1cm, etc.

From what i found online.. merging the 3 new indexers into the old cluster seems to be the preferred method.  Does anyone have a link to a detailed writeup on how to do so with all the little nuances comes with it?  are differing splunk versions okay?  do i change rep factor? im sure there are a bunch of steps to this method.

I appreciate any help!

[richgalloway]

I second what @_JP said.  The situation is complicated by the two different environments.  Having on-prem and AWS indexers in the same cluster would quickly run up your AWS data export charges unless you put the old indexers in manual detention to keep them from accepting replicated data. 

PS should be able to help.

[isoutamo]

Hi

as other already said this is doable. Here is my old post how to do it https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platf...

Of course in your case you must take care of connection between those sites. That must be enough fast to get this working.

One other notice is that you must have same splunk version on all nodes in cluster. So you must select between update old first or install old version to AWS first.

Based on amount of your data and connection speed between sites you could do this or other option is just create a new idx cluster on AWS and after it is in use then create additional (temporary) cluster for old onprem data and add it as a second cluster for your search heads. That way you could utilise e.g. snowball to transfer data from onsite to AWS (if your connection cannot handle it with reasonable time).  

r. Ismo

[_JP]

Yes, this is possible.  I don't know of any write-up on how to do this.  The simplest answer that you'll receive is, "This is a great thing for Splunk PS to help with..."

That being said, I feel like you are on the right track - you can add the new indexers to the cluster, force a rebalance/replication to the new ones, then shutdown the old ones.  

Do you have plenty of hardware to POC this in a non-production environment?  I would practice this in not-production, and then once you are able to do it successfully you can repeat in prod.  Also, do you have backups in case anything goes wrong?

(Also, @Darthsplunker showing up with a hard-mode question as a brand new community user 😂  I applaud and salute you!)