Getting Data In

How to migrate buckets from a standalone indexer to a multisite cluster environment?

sylim_splunk
Splunk Employee
Splunk Employee

I had an older standalone splunk indexer. I set up a new multisite cluster (2 indexers, site rep/search factor of 2) and have all data available at both sites. Splunk version is the latest, 7.1.2.1.
I want to take the old data (legacy indexer, not replicated), and have it replicate in the new multisite cluster.

I am dealing with ~1tb of logs across many years. Just adding the old indexer to my search heads is not a valid workaround -- 100% data availability at both sites is my primary concern.

  • Steps followed - 1.Set up an indexer (single_idx) with index named "test". 2.Set up an indexer cluster (multi_idx1, multi_idx_2) with index "test". 3.Copy a warm bucket from single_idx to multi_idx_1 4.Rename the bucket to append the multi_idx_1 guid following the clustered bucket naming convention. 5.Watch the multi_idx_2 db folder and watch for a rb_ folder corresponding to the manually added bucket. It never replicates to the other site indexer.
0 Karma
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

As of this writing where the latest version is 7.1.2.1, this is not a supported feature.
The steps you have done works for the case of the migration from standalone to single site cluster. But it is not working for the case where you migrate to multisite cluster as the buckets created in single site cluster/standalone don't have site information.
This will be implemented and available in 7.2+, where a parameter, "constrain_singlesite_buckets" is available in server.conf.

View solution in original post

0 Karma

oliverj
Communicator

With the release of 7.2, I see this in the new server.conf:

constrain_singlesite_buckets =
* Only valid for mode=master and is only used if multisite is true.
* Specifies whether the cluster keeps single-site buckets within one site
in multisite clustering.
* When this setting is "true", buckets in a single site cluster do not
replicate outside of their site. The buckets follow 'replication_factor'
'search factor' policies rather than 'site_replication_factor'
'site_search_factor' policies. This is to mimic the behavior of
single-site clustering.
* When this setting is "false", buckets in non-multisite clusters can
replicate across sites, and must meet the specified
'site_replication_factor' and 'site_search_factor' policies.
* Default: true

0 Karma

oliverj
Communicator

Update: This worked.
Added "constrain_singlesite_buckets = false" to the server.conf on my cluster master.

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee

Thanks for trying it out.

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee

As of this writing where the latest version is 7.1.2.1, this is not a supported feature.
The steps you have done works for the case of the migration from standalone to single site cluster. But it is not working for the case where you migrate to multisite cluster as the buckets created in single site cluster/standalone don't have site information.
This will be implemented and available in 7.2+, where a parameter, "constrain_singlesite_buckets" is available in server.conf.

0 Karma

mibrown_splunk
Splunk Employee
Splunk Employee

Also worth a comment that pre-7.2, you could still migrate the buckets from standalone to multisite cluster. The difference pre-7.2 is that the buckets will replicate but only within the site the buckets were sent to, based on the single-site replication settings. Like you said, due to the new parameter in 7.2, you get to replicate buckets across the multisite environment as though they were generated natively.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...