This message may go away if waiting for longer time. However, I don't want to wait and any command to roll all the bucket.
You can run below search on your cluster master to get a list of bucket that have status="bucket hasn't rolled yet"
| rest splunk_server=local /services/cluster/master/fixup level=replication_factor
| table title, latest.reason
| rename latest.reason AS LatestReason
| rename totle AS bucketID
| regex LatestReason="bucket hasn't rolled yet" | table buckekID
Once you got the bucketId, simple run below command on your Cluster Master will roll the bucket.
curl -k -u admin:changme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets -d "bucket_id=<BUCIET_ID>”
For example,
curl -k -u admin:changeme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets -d "bucket_id=_internal~4520~11111111-1111-1111-1111-111111111111”
IF you prefer, you can do this in batch mode.
e.g. cat /var/tmp/bucketId.txt
os~100~2FC3562D-9D2D-49CB-A598-89E47397E5D7
_internal~4523~11111111-1111-1111-1111-111111111111
for i in `cat /var/tmp/bucketId.txt`; do curl -k -u admin:changeme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets -d "bucket_id=$i"; done
This is a great answer, even though it didn't work for me. For some reason, I constantly have hundreds of such bucket status that just didn't go away, even after restarting the CM and rolling-restart of my 7 indexers. When I ran this in batch mode, I had the following error message for every of the listed 685 buckets:
Cannot roll a bucket with bid=ciscolog~330~5442CDRE-2540-45C7-DD32-23D48D394D8F to warm, Reason="Master already has committed size for this bucket."
Any clue how to resolve this?
Did you try enable the maintenance mode and then restart the CM?
I went into the CM and enable maintenance mode, after which I then do a "splunk restart". However, it's still the same error. Any pointers will be appreciated.