topic Re: How do I migrate Indexes from one multisite cluster to another multisite cluster? in Getting Data In

How do I migrate Indexes from one multisite cluster to another multisite cluster?

harsmarvania57 — Thu, 24 May 2018 09:57:51 GMT

Hi Community Members,

I would like to migrate indexes from one multisite cluster to another multisite cluster. Both the multisite cluster have same RF/SF. What will be the steps for this ?

I have gone through few of the answers on community and try to find out documentation for multisite index migration but no luck.

My understanding is
1.) Copy buckets from db and colddb directories from old multisite cluster to new multisite cluster
2.) Check any bucket ID conflict and if so rename those bucket with newer ID

Now questions are
1.) Do I need to copy buckets starting with rb_timestamp_timestamp_ID_GUID ?
2.) Do I need to copy <index name>.dat file ?

Thanks,
Harshil

Re: How do I migrate Indexes from one multisite cluster to another multisite cluster?

adonio — Thu, 24 May 2018 14:52:05 GMT

hello there,

before going any further with this, do you have to move the indexes?
what is the goal of this data move? what purpose does it serves?
imho there are almost always better ways to achieve of the goals behind the need for moving data (indexes) especially clustered ones, without the pain of moving the data.

Re: How do I migrate Indexes from one multisite cluster to another multisite cluster?

harsmarvania57 — Thu, 24 May 2018 15:49:47 GMT

Hi @adonio,

Yes, I would like to move the indexes. I am in process to move application data from 1st multisite cluster to 2nd multisite cluster due to security reason. And once application data will be separated from 1st multisite cluster, both the clusters will not have any relationship. Even data will be searched from different SHC and those SHC will search data from their respective multisite indexer cluster.

I know that to move data in clustered environment is bit tricky but I am still looking for good solution if anyone have.

Re: How do I migrate Indexes from one multisite cluster to another multisite cluster?

dxu_splunk — Tue, 29 Sep 2020 19:38:01 GMT

1) make sure theres no GUID conflicts between the clusters - otherwise if we move buckets from one cluster to another that has indexers with the same GUID, it could conflict with an existing bucket.

2) do the clusters use the same available_sites and site namings?
if not, we'll need to configure to use the site_mappings settings before we move buckets from one multisite to another multisite. please see https://docs.splunk.com/Documentation/Splunk/7.1.1/Indexer/Decommissionasite#Syntax
(the documentation is in decommissioning a site, but the concept is similar - buckets that originated from siteA that are no longer available need to get a new "originSite", because it has to satisfy the originSite policy - we cant get any copies on siteA anymore so we'll map it to a new site)

3) when copying over the data, you can copy just the db_ versions of the bucket, but it'll probably be faster (and more correct) to copy over everything (db_ and rb_) of the index, (*more correct because its possible a bucket only has rb_ copies and no db_ copy)

Re: How do I migrate Indexes from one multisite cluster to another multisite cluster?

harsmarvania57 — Fri, 25 May 2018 12:13:38 GMT

Hi @dxu [Splunk],

Thanks for your valuable time.

Yes, both clusters use the same available_sites and site namings.

Re: How do I migrate Indexes from one multisite cluster to another multisite cluster?

vadivel_parames — Wed, 22 Jan 2020 13:55:39 GMT

Hi, @harsmarvania57, i have a similar requirement. Can you provide me the steps how to migrate the date from one multi-site index cluster environment to an another multi-site environment? I have 2 sites, each site has 3 peers (total six peers) in one environment and the same configuration in an another environment. Should I copy buckets from all the indexers in the old environment to all the indexers in the new environment?