Getting Data In
Highlighted

How to migrate buckets from a standalone indexer to a multisite cluster environment?

Splunk Employee
Splunk Employee

I had an older standalone splunk indexer. I set up a new multisite cluster (2 indexers, site rep/search factor of 2) and have all data available at both sites. Splunk version is the latest, 7.1.2.1.
I want to take the old data (legacy indexer, not replicated), and have it replicate in the new multisite cluster.

I am dealing with ~1tb of logs across many years. Just adding the old indexer to my search heads is not a valid workaround -- 100% data availability at both sites is my primary concern.

  • Steps followed - 1.Set up an indexer (singleidx) with index named "test". 2.Set up an indexer cluster (multiidx1, multiidx2) with index "test". 3.Copy a warm bucket from singleidx to multiidx1 4.Rename the bucket to append the multiidx1 guid following the clustered bucket naming convention. 5.Watch the multiidx2 db folder and watch for a rb folder corresponding to the manually added bucket. It never replicates to the other site indexer.
0 Karma
Highlighted

Re: How to migrate buckets from a standalone indexer to a multisite cluster environment?

Splunk Employee
Splunk Employee

As of this writing where the latest version is 7.1.2.1, this is not a supported feature.
The steps you have done works for the case of the migration from standalone to single site cluster. But it is not working for the case where you migrate to multisite cluster as the buckets created in single site cluster/standalone don't have site information.
This will be implemented and available in 7.2+, where a parameter, "constrainsinglesitebuckets" is available in server.conf.

View solution in original post

0 Karma
Highlighted

Re: How to migrate buckets from a standalone indexer to a multisite cluster environment?

Splunk Employee
Splunk Employee

Also worth a comment that pre-7.2, you could still migrate the buckets from standalone to multisite cluster. The difference pre-7.2 is that the buckets will replicate but only within the site the buckets were sent to, based on the single-site replication settings. Like you said, due to the new parameter in 7.2, you get to replicate buckets across the multisite environment as though they were generated natively.

0 Karma
Highlighted

Re: How to migrate buckets from a standalone indexer to a multisite cluster environment?

Communicator

With the release of 7.2, I see this in the new server.conf:

constrainsinglesitebuckets =
* Only valid for mode=master and is only used if multisite is true.
* Specifies whether the cluster keeps single-site buckets within one site
in multisite clustering.
* When this setting is "true", buckets in a single site cluster do not
replicate outside of their site. The buckets follow 'replicationfactor'
'search factor' policies rather than 'site
replicationfactor'
'site
searchfactor' policies. This is to mimic the behavior of
single-site clustering.
* When this setting is "false", buckets in non-multisite clusters can
replicate across sites, and must meet the specified
'site
replicationfactor' and 'sitesearch_factor' policies.
* Default: true

0 Karma
Highlighted

Re: How to migrate buckets from a standalone indexer to a multisite cluster environment?

Communicator

Update: This worked.
Added "constrainsinglesitebuckets = false" to the server.conf on my cluster master.

0 Karma
Highlighted

Re: How to migrate buckets from a standalone indexer to a multisite cluster environment?

Splunk Employee
Splunk Employee

Thanks for trying it out.

0 Karma