Knowledge Management

[SmartStore]Role of file .buckets_synced_to_remote_storage in migration from local to SmartStore.

rbal_splunk
Splunk Employee
Splunk Employee

Lets assume that I have 20 node Indexer cluster deployments. Enabled smart store and migration started but due to a network issue the migration was interrupted. We are trying to figure out the best bay to restart a migration. How can that be done? What is role of file .buckets_synced_to_remote_storage in migration?

Tags (1)
0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

At startup , if an index is s2-enabled, splunk check to see if bucket need to be uploaded. To check if buckets need to be uploade splunk look for file $SPLUNK_DB/var/lib/splunk//db/.buckets_synced_to_remote_storage exists. The presence of this file indicates that splunk don't need to upload files to the remote storage and therefore no migration needs to happen.

At startup there are no buckets on the system , or creation of new index.

During migration upload to the remote storage. A migration upload will start on S2 enabled indexes if there are buckets in the index. The "migration upload" will first do a bulk-add of all existing buckets to the CacheManger. The buckets will be added in randomized order. Then it will touch a $SPLUNK_DB/var/lib/splunk/.buckets_synced_to_remote_storage file, and then it will do registerExisting on the CacheManager, which will trigger the upload jobs to begin.

Before uploading each individual bucket the CacheManager will check if that bucket exists on the remote storage first.

example:

find  $SPLUNK_DB -type f -name .buckets_synced_to_remote_storage
./var/lib/splunk/audit/db/.buckets_synced_to_remote_storage
./var/lib/splunk/_internaldb/db/.buckets_synced_to_remote_storage
./var/lib/splunk/_introspection/db/.buckets_synced_to_remote_storage

In case the migration had to be restart, you will need to shutdown the indexers delete .buckets_synced_to_remote_storage from all indexer. Upon restart migration will restart

0 Karma

mdoyle_splunk
Splunk Employee
Splunk Employee

Do NOT remove this file unless you also change the indexer GUID using splunk clone-prep-clear-config or it will possible corrupt previously uploaded buckets that have been evicted from the local cache.

clone-prep-clear-config is safe to use because it emulates a new indexer being added to the cluster and it will not upload buckets that already exist that were uploaded by a different indexer.

If there is any possibility that your data on the indexer is not 100% complete due to eviction, you MUST change the guid when removing .buckets_synced_to_remote_storage

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...