This error is for hot buckets that are being index on the Originating node - but the issue is Data is not being replicated to any other Node to meet the required Replication and Search Factor.
This Data will still be searchable from the Originating node- but if the Originating Nodes goes down the data will not be available on any other node.
These error messages will disappears as these these buckets will roll from hot to warm. Upon roll of bucket Cluster Master will check if the RF and SF are not met - it will force copy of bucket to meet RF and SF.
It may take days for bucket to roll - In case you need to address this issue right away you have few options.
1) Easy approach will be to issue rolling restart from Cluster the hot buckets will roll to warm and the issue wil lget addressed.
or
2) Other option will be to look for index have buckets in-such state and issue the command that will roll the hot bucket .Ran the following curl command on internal index _audit and roll the buckets from hot to warm
curl -k -u admin:changeme https://HOST:PORT/services/data/indexes/YOUR_INDEX/roll-hot-buckets -X POST
HOST is the StreamingSource server name, PORT being 8089, YOUR_INDEX is the index needing to roll
example of _audit
curl -k -u admin:PASSWORD https://idx2.coxauto.splunkcloud.com:8089/services/data/indexes/_audit/roll-hot-buckets -X POST
You will need to run this for each of teh index with buckets in run on remaining indexes found here:
