I have Index cluster with 3 Nodes and intermittently I see that ‘Search Factor” and “Replication Factor” not met.
When we navigate to setting>Clustering> Buckets> Bucket fix and see that many bucket show status like “Cannot replicate as bucket hasn’t rolled yet.”
This error is for hot buckets that are being index on the Originating node - but the issue is Data is not being replicated to any other Node to meet the required Replication and Search Factor.
This Data will still be searchable from the Originating node- but if the Originating Nodes goes down the data will not be available on any other node.
These error messages will disappears as these these buckets will roll from hot to warm. Upon roll of bucket Cluster Master will check if the RF and SF are not met - it will force copy of bucket to meet RF and SF.
It may take days for bucket to roll - In case you need to address this issue right away you have few options.
1) Easy approach will be to issue rolling restart from Cluster the hot buckets will roll to warm and the issue wil lget addressed.
2) Other option will be to look for index have buckets in-such state and issue the command that will roll the hot bucket .Ran the following curl command on internal index _audit and roll the buckets from hot to warm
Newer versions of Splunk (at least in 7.x) have a roll bucket button under the "bucket status" on the indexing cluster page. Or you could just wait until they roll to warm if you are not worried about the replication not occurring until then...
Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/ Version Control for Splunk https://splunkbase.splunk.com/app/4355/