Lets assume that I have 20 node Indexer cluster deployments. Enabled smart store and migration started but due to a network issue the migration was interrupted. We are trying to figure out the best bay to restart a migration. How can that be done? What is role of file .bucketssyncedtoremotestorage in migration?
At startup , if an index is s2-enabled, splunk check to see if bucket need to be uploaded. To check if buckets need to be uploade splunk look for file $SPLUNKDB/var/lib/splunk//db/.bucketssyncedtoremote_storage exists. The presence of this file indicates that splunk don't need to upload files to the remote storage and therefore no migration needs to happen.
At startup there are no buckets on the system , or creation of new index.
During migration upload to the remote storage. A migration upload will start on S2 enabled indexes if there are buckets in the index. The "migration upload" will first do a bulk-add of all existing buckets to the CacheManger. The buckets will be added in randomized order. Then it will touch a $SPLUNKDB/var/lib/splunk/.bucketssyncedtoremote_storage file, and then it will do registerExisting on the CacheManager, which will trigger the upload jobs to begin.
Before uploading each individual bucket the CacheManager will check if that bucket exists on the remote storage first.
find $SPLUNK_DB -type f -name .buckets_synced_to_remote_storage
In case the migration had to be restart, you will need to shutdown the indexers delete .bucketssyncedtoremotestorage from all indexer. Upon restart migration will restart