[SmartStore]Increase RF=2,SF=1 to RF=SF=2 and cluster took long time to meet RF, SF and Mark All Data Searchable.

[rbal_splunk]

@We have two node Cluster using smartstore

@Initially configured as RF=2 and Sf=1 and CM's user interface shows ( "All Data is Searchable", "Search Factor is Met" and "Replication Factor is Met")
@We change RF=SF=2 and restarted CM and post restart the CM tool very long time to meet "All Data is Searchable" to Turn Green.
@These two nodes have about 2000 buckets each.

As per the documentation with smarstore the RF and SF should meet much faster

[rbal_splunk]

Analysis of the log files reveals following, for this analysis focused on Bucket "docker_main~827~32226E90-F8B0-4E01-9A5C-54CB63AD5BDC|" that was stuck in the fixup queue for a long time.

Just before changing the SF=2 bucket "docker_main~827~32226E90-F8B0-4E01-9A5C-54CB63AD5BDC|" got moved to frozen by bucket mover, so it will result in deletion of the bucket.

CM got restarted just after this and action was not completed on other nodes but the bucket got removed from the cache manager as well.

Since SF became 2 Splunk ended up doing fix-up tasks on all the available buckets at this moment.

Eventually, it does a bucket rebuild job to replicate this bucket in other indexers as well as part of the fix-up job.

11-28-2018 18:11:07.545 -0500 INFO CMRepJob - Finished CMResyncBucketJob, bid=docker_main~827~32226E90-F8B0-4E01-9A5C-54CB63AD5BDC guid=FD5C20E7-FE58-47F5-B2CD-BD7C4E5EE029 _rc=1 hasBucketBeforeResync=1 hasBucketAfterResync=1

Eventually, the bucket gets frozen again and gets deleted