Deployment Architecture
Highlighted

Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

Communicator

Hello,

Actually, we have 6 indexers and 1 SH. We are migrating our platefrom to a multisite indexer cluster with 18 indexers and a search head cluster (3SH).

In legacy platefrom, we have more that 20TB stored by indexer.

We will perform these steps to migrate indexers :

1-Transform bucket on old indexers, from standalone to cluster bucket (via script)
2-Add old indexers to the cluster
3-Put the old indexers into Detention : Issue a 'splunk offline --enforce-counts' command to ONE old indexer
4-Wait for the buckets to migrate off the old indexer.
5-Repeat steps 3-4 for the remaining old indexers
6-Once all buckets are moved to the new indexers, remove the old indexers from the cluster

I'm afraid that copying many TB between two sites, causes network congestion. Do you have any ideas on best practices doing that ? Is there a throttling replication parameter that we can use ? Is there another optimized way ? Is the steps we follow are good ?

Thanks in advance.

Highlighted

Re: Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

SplunkTrust
SplunkTrust

Hi @Afef,

Which version of Splunk are you running ? As of 7.2 you can use the constrain_singlesite_buckets setting to have the buckets migrated automatically after converting the CM to multi-site.
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Migratetomultisite

If you're on a version prior to 7.2 then I advise you to have a look here for some ideas on how the migration can be done: https://answers.splunk.com/answers/376054/what-could-be-the-best-approach-to-migrate-an-exis.html

Your steps will do the trick but "I'm afraid that copying many TB between two sites, causes network congestion." will happen for sure and there is no throttling that can be configured from the Splunk side. To be on the safe side you could ask the network team to configure a BW limit for your Splunk-Splunk communication to avoid overusing during the replication.

Cheers,
David

Highlighted

Re: Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

Communicator

Hi David,
thanks for your answer.

yes we are on Splunk latest version, i'm aware of constrainsinglesitebuckets new parameter 🙂 (Our indexers are note in cluster actually, that's why we shoold to transfrom the buckets to cluster buckets before, and then add the constrainsinglesitebuckets to allow the intersite replication of old buckets).

0 Karma
Highlighted

Re: Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

SplunkTrust
SplunkTrust

Oh, I read you have 6 indexers assumed you're in single site, this is going to be fun ^^

0 Karma
Highlighted

Re: Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

Contributor

Hey guys,
In my case I'm migrating from ONE Pre 7 version indexer to THREE Post 7 version indexers cluster.

Am I going to have difficulties?
Should I upgrade my only indexer first?

0 Karma
Highlighted

Re: Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

Super Champion

There are two parts to your query
1. Indexer cluster migration
2. Search Head cluster

(1) Indexer cluster migration, at a high level what you have said should be good. But be very cautious to backup data in case if some corruption happens.

Does your whole data is 20TB? If that's the case, its not too much. Replication would finish quite easily with 18indexers. How much is your network bandwidth for site to site? but normally 10GB networks can finish this very quickly

As a Plan C; Another option is to re-index the data if need be. It is 20TB and you could have a word with Splunk to see if blowing license for a day is a problem or not. I know it is bit clunky, but you can save migration and the data will be indexed correctly afresh.

(2) Search Head cluster is a different ball game. You may find it difficult , if you are doing it first time. WE normally make the apps modular, so just a copy-paste of relevant apps will create SHC and join to indexer cluster.

Ensure you have deployer and cluster master for the SHC & indexer cluster respectively

0 Karma
Highlighted

Re: Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

Esteemed Legend

Also (3) Data bucket conversion

0 Karma
Highlighted

Re: Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

Communicator

Hello, thanks for your answer.
we have 20TB by indexer, so 120TB in total to be migrated. I think it will take a while to be replicated ...

0 Karma
Highlighted

Re: Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

Esteemed Legend

It will not replicate if you don't manually convert the buckets. Old buckets are not compatible with replication.

0 Karma
Highlighted

Re: Best practices Migrating from standalone indexers to a multisite indexer cluster and a search head cluster

Contributor

Hello!
"Old buckets" = "all previously indexed data" , right?
I.e. only new data is going to be replicated, old data will be accessible and can be searched anyway. Right?

Best regards.

0 Karma