Index Cluster - Search Factro and Replication Factor NOt Met - Error “Cannot replicate as bucket hasn’t rolled yet.”

[sat94541]

I have Index cluster with 3 Nodes and intermittently I see that ‘Search Factor” and “Replication Factor” not met.
When we navigate to setting>Clustering> Buckets> Bucket fix and see that many bucket show status like “Cannot replicate as bucket hasn’t rolled yet.”

How to resolve this issue and what is the impact?

[rbal_splunk]

This error is for hot buckets that are being index on the Originating node - but the issue is Data is not being replicated to any other Node to meet the required Replication and Search Factor.

This Data will still be searchable from the Originating node- but if the Originating Nodes goes down the data will not be available on any other node.

These error messages will disappears as these these buckets will roll from hot to warm. Upon roll of bucket Cluster Master will check if the RF and SF are not met - it will force copy of bucket to meet RF and SF.

It may take days for bucket to roll - In case you need to address this issue right away you have few options.

1) Easy approach will be to issue rolling restart from Cluster the hot buckets will roll to warm and the issue wil lget addressed.
or
2) Other option will be to look for index have buckets in-such state and issue the command that will roll the hot bucket .Ran the following curl command on internal index _audit and roll the buckets from hot to warm

curl -k -u admin:changeme https://HOST:PORT/services/data/indexes/YOUR_INDEX/roll-hot-buckets -X POST

HOST is the StreamingSource server name, PORT being 8089, YOUR_INDEX is the index needing to roll

example of _audit

curl -k -u admin:PASSWORD https://idx2.coxauto.splunkcloud.com:8089/services/data/indexes/_audit/roll-hot-buckets -X POST

You will need to run this for each of teh index with buckets in run on remaining indexes found here:

[kishor_pinjark2]

On CM if I see there are more than 10 indexes having this issue, then I have to run below command for all indexes ?

curl -k -u admin:changeme https://HOST:PORT/services/data/indexes/YOUR_INDEX/roll-hot-buckets -X POST

[gjanders]

Newer versions of Splunk (at least in 7.x) have a roll bucket button under the "bucket status" on the indexing cluster page. Or you could just wait until they roll to warm if you are not worried about the replication not occurring until then...

[sudosplunk]

Yes. If you decide to manually roll buckets, then you have to run the command for each index separately (10 times per you question).