Deployment Architecture

Index Cluster - Search Factro and Replication Factor NOt Met - Error “Cannot replicate as bucket hasn’t rolled yet.”

sat94541
Communicator

I have Index cluster with 3 Nodes and intermittently I see that ‘Search Factor” and “Replication Factor” not met.
When we navigate to setting>Clustering> Buckets> Bucket fix and see that many bucket show status like “Cannot replicate as bucket hasn’t rolled yet.”

alt text

How to resolve this issue and what is the impact?

Tags (3)
0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

This error is for hot buckets that are being index on the Originating node - but the issue is Data is not being replicated to any other Node to meet the required Replication and Search Factor.

This Data will still be searchable from the Originating node- but if the Originating Nodes goes down the data will not be available on any other node.

These error messages will disappears as these these buckets will roll from hot to warm. Upon roll of bucket Cluster Master will check if the RF and SF are not met - it will force copy of bucket to meet RF and SF.

It may take days for bucket to roll - In case you need to address this issue right away you have few options.

1) Easy approach will be to issue rolling restart from Cluster the hot buckets will roll to warm and the issue wil lget addressed.
or
2) Other option will be to look for index have buckets in-such state and issue the command that will roll the hot bucket .Ran the following curl command on internal index _audit and roll the buckets from hot to warm

curl -k -u admin:changeme https://HOST:PORT/services/data/indexes/YOUR_INDEX/roll-hot-buckets -X POST

HOST is the StreamingSource server name, PORT being 8089, YOUR_INDEX is the index needing to roll

example of _audit

curl -k -u admin:PASSWORD https://idx2.coxauto.splunkcloud.com:8089/services/data/indexes/_audit/roll-hot-buckets -X POST

You will need to run this for each of teh index with buckets in run on remaining indexes found here:

kishor_pinjark2
Path Finder

On CM if I see there are more than 10 indexes having this issue, then I have to run below command for all indexes ?

curl -k -u admin:changeme https://HOST:PORT/services/data/indexes/YOUR_INDEX/roll-hot-buckets -X POST

0 Karma

gjanders
SplunkTrust
SplunkTrust

Newer versions of Splunk (at least in 7.x) have a roll bucket button under the "bucket status" on the indexing cluster page. Or you could just wait until they roll to warm if you are not worried about the replication not occurring until then...

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

sudosplunk
Motivator

Yes. If you decide to manually roll buckets, then you have to run the command for each index separately (10 times per you question).

Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.