Does not meet: primacy & sf & rf

[blanky]

There was a time when the indexer server shut down unexpectedly, 

And I've been struggle with indexer clustering rf & sf were doesn't meet.

Every index are satisfied with rf & sf, but only one index doesn't meet sf & rf

I have tried roll / resync / rolling restart in the Master node, but it doesn't work.

I'm trying to find the error bucket and remove it from the CLI environment, and restart the cluster.

Is it right solution to solve this problem?? 
Or Suggest me the better way to solve it. please

[kiran_panchavat]

@blanky 

I kindly request you to raise a support ticket for further troubleshooting. You may refer to the details below if they are helpful.

Search Factor and Replication Factor is not met on Cluster Manager | Splunk

[kiran_panchavat]

@blanky 

You can run below search on your cluster master to get a list of bucket that have status="bucket hasn't rolled yet"

| rest splunk_server=local /services/cluster/master/fixup level=replication_factor 
| table title, latest.reason
| rename latest.reason AS LatestReason
| rename totle AS bucketID
| regex LatestReason="bucket hasn't rolled yet" | table buckekID

Once you got the bucketId, simple run below command on your Cluster Master will roll the bucket.

curl -k -u admin:changme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets -d "bucket_id=<BUCIET_ID>”

For example,

curl -k -u admin:changeme https://localhost:8089/services/cluster/master/control/control/roll-hot-buckets -d "bucket_id=_internal~4520~11111111-1111-1111-1111-111111111111”

[kiran_panchavat]

@blanky 

look at the cluster master’s logs (splunkd.log) for errors related to the problematic index.

[kiran_panchavat · Accepted solution]

@blanky 

If the RF and SF are not met then everything is not fine with your indexers.  Until the RF is met a failure of an indexer could result in data loss.

Since it looks like all indexers are up, it should just be a matter of waiting for buckets to be replicated.

Check to see if any buckets are stuck in fixup tasks? If so, resolve issue.

indexer clustering > Indexes > Bucket Status

[blanky]

bucket status is "Waiting 'target_wait_time' before replicating buckt".

but, i've been waiting for a week for replicating. but it doesn't changed.

[kiran_panchavat]

@blanky 

If the bucket status is stuck on "Waiting 'target_wait_time' before replicating bucket" for a week, that’s a clear sign something’s gone wrong in your indexer clustering setup.

Ensure there are no network issues between the indexers. Network problems can impede bucket replication.

Check this documentation for more information: Bucket replication issues - Splunk Documentation 

[blanky]

Thank you for your support.

I found a error bucket in the bucket state, removed it directly from the CLI environment, and rebooted it to fix the problem.

And the rf & sf met finally. Bucket with error was not created in folder form, but in file form.