Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

[Afef]

Hello,

Actually, we have 6 indexers and 1 SH. We are migrating our platefrom to a multisite indexer cluster with 18 indexers and a search head cluster (3SH).

In legacy platefrom, we have more that 20TB stored by indexer.

We will perform these steps to migrate indexers :

1-Transform bucket on old indexers, from standalone to cluster bucket (via script)
2-Add old indexers to the cluster
3-Put the old indexers into Detention : Issue a 'splunk offline --enforce-counts' command to ONE old indexer
4-Wait for the buckets to migrate off the old indexer.
5-Repeat steps 3-4 for the remaining old indexers
6-Once all buckets are moved to the new indexers, remove the old indexers from the cluster

I'm afraid that copying many TB between two sites, causes network congestion. Do you have any ideas on best practices doing that ? Is there a throttling replication parameter that we can use ? Is there another optimized way ? Is the steps we follow are good ?

Thanks in advance.

[MuS]

Hi Afef,

this looks all good and there is no way to throttling the replication except some network layer based QoS or other magic. You also forgot step 7: Have either a flight ticket or Whisky ready depending on the end result 🙂

cheers, MuS

[koshyk]

There are two parts to your query
1. Indexer cluster migration
2. Search Head cluster

(1) Indexer cluster migration, at a high level what you have said should be good. But be very cautious to backup data in case if some corruption happens.

Does your whole data is 20TB? If that's the case, its not too much. Replication would finish quite easily with 18indexers. How much is your network bandwidth for site to site? but normally 10GB networks can finish this very quickly

As a Plan C; Another option is to re-index the data if need be. It is 20TB and you could have a word with Splunk to see if blowing license for a day is a problem or not. I know it is bit clunky, but you can save migration and the data will be indexed correctly afresh.

(2) Search Head cluster is a different ball game. You may find it difficult , if you are doing it first time. WE normally make the apps modular, so just a copy-paste of relevant apps will create SHC and join to indexer cluster.

Ensure you have deployer and cluster master for the SHC & indexer cluster respectively

[Afef]

Hello, thanks for your answer.
we have 20TB by indexer, so 120TB in total to be migrated. I think it will take a while to be replicated ...

[woodcock]

It will not replicate if you don't manually convert the buckets. Old buckets are not compatible with replication.

[highsplunker]

Hello!
"Old buckets" = "all previously indexed data" , right?
I.e. only new data is going to be replicated, old data will be accessible and can be searched anyway. Right?

Best regards.

[DavidHourani]

You're right ! Only "new" buckets created in the cluster will be cluster aware.

Also make sure to build your cluster as a multi-site cluster even if you will only be using one site. It will make the buckets site aware in case of any future migrations.

Let me know if you need more details.

[highsplunker]

Ok thank you friends! There's a lot of work to do for me. I'm only preparing now.

[DavidHourani]

good luck m8

[woodcock]

Yes, exactly.

[woodcock]

Also (3) Data bucket conversion

[DavidHourani]

Hi @Afef,

Which version of Splunk are you running ? As of 7.2 you can use the constrain_singlesite_buckets setting to have the buckets migrated automatically after converting the CM to multi-site.
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Migratetomultisite

If you're on a version prior to 7.2 then I advise you to have a look here for some ideas on how the migration can be done: https://answers.splunk.com/answers/376054/what-could-be-the-best-approach-to-migrate-an-exis.html

Your steps will do the trick but "I'm afraid that copying many TB between two sites, causes network congestion." will happen for sure and there is no throttling that can be configured from the Splunk side. To be on the safe side you could ask the network team to configure a BW limit for your Splunk-Splunk communication to avoid overusing during the replication.

Cheers,
David

[Afef]

Hi David,
thanks for your answer.

yes we are on Splunk latest version, i'm aware of constrain_singlesite_buckets new parameter 🙂 (Our indexers are note in cluster actually, that's why we shoold to transfrom the buckets to cluster buckets before, and then add the constrain_singlesite_buckets to allow the intersite replication of old buckets).

[DavidHourani]

Oh, I read you have 6 indexers assumed you're in single site, this is going to be fun ^^

[highsplunker]

Hey guys,
In my case I'm migrating from ONE Pre 7 version indexer to THREE Post 7 version indexers cluster.

Am I going to have difficulties?
Should I upgrade my only indexer first?