SF & RF doesn't meet after removing indexer from cluster

[sandeepreddy947]

I have a bucket in fixup tasks in indexer cluster-> bucket status, its been struck.  Both SF & RF. So, both SF and RF are not met in indexer cluster. 

I tried to roll and resync bucket manually, that didn't work. There're no buckets in excess buckets, i've cleared them like more than 3hrs.

Is there any way to meet SF & RF without loosing data or bucket ? I even tried to restart Splunk process on that Indexer

Forgot to mention, i had a /opt/cold drive that has I/O error on an indexer. To get it fix i had stop Splunk and remove an indexer from indexer cluster, All other indexers are up and running since last night.  All 45 indexers in cluster-master are up and running and left it to bucket fixup tasks to fix and it also to rebalance overnight. When i check morning there're only 2 fixup tasks left one is in SF & one in RF. 

Does it also need manual data rebalance to perform from indexer-cluster as well ?

[isoutamo]

Hi

as you have had some I/O errors on your /opt/cold there is possibility that there are some buckets which are corrupted and cannot used anymore. You should find from _internal -log what cause that issue. Just search those buckets from it which you have on MC's view of SF&RF not met and in fixing task.

After you have identified those reasons you could decide how to proceed. Maybe just remove primary bucket and use your replicas or something else, but this is totally dependent on the reason what you found from internal.

What are your SF & RF and have you single site or multisite cluster?

Basically it should't need a data rebalancing unless your bucket count has totally unbalanced between indexers. You could see that e.g. via REST calls.

r. Ismo

[gcusello]

Hi @sandeepreddy947,

having an infrastructure like your (45 Indexers), the only thing is to open a ticket to Splunk Support.

ciao.

Giuseppe