Cluster Master have SF/RF not meet due to a lot of bucket got status 'bucket hasn't rolled yet'

[daniel_splunk]

This message may go away if waiting for longer time. However, I don't want to wait and any command to roll all the bucket.

[daniel_splunk]

You can run below search on your cluster master to get a list of bucket that have status="bucket hasn't rolled yet"

| rest splunk_server=local /services/cluster/master/fixup level=replication_factor 
| table title, latest.reason
| rename latest.reason AS LatestReason
| rename totle AS bucketID
| regex LatestReason="bucket hasn't rolled yet" | table buckekID

Once you got the bucketId, simple run below command on your Cluster Master will roll the bucket.

curl -k -u admin:changme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets  -d "bucket_id=<BUCIET_ID>”

For example,
curl -k -u admin:changeme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets  -d "bucket_id=_internal~4520~11111111-1111-1111-1111-111111111111”

IF you prefer, you can do this in batch mode.

e.g. cat /var/tmp/bucketId.txt
os~100~2FC3562D-9D2D-49CB-A598-89E47397E5D7
_internal~4523~11111111-1111-1111-1111-111111111111

for i in `cat /var/tmp/bucketId.txt`; do curl  -k -u admin:changeme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets -d "bucket_id=$i"; done

[smallfry]

This is a great answer, even though it didn't work for me. For some reason, I constantly have hundreds of such bucket status that just didn't go away, even after restarting the CM and rolling-restart of my 7 indexers. When I ran this in batch mode, I had the following error message for every of the listed 685 buckets:

Cannot roll a bucket with bid=ciscolog~330~5442CDRE-2540-45C7-DD32-23D48D394D8F to warm, Reason="Master already has committed size for this bucket."

Any clue how to resolve this?

[daniel_splunk]

Did you try enable the maintenance mode and then restart the CM?

[smallfry]

I went into the CM and enable maintenance mode, after which I then do a "splunk restart". However, it's still the same error. Any pointers will be appreciated.