Deployment Architecture

SF & RF doesn't meet after removing indexer from cluster

sandeepreddy947
Path Finder

I have a bucket in fixup tasks in indexer cluster-> bucket status, its been struck.  Both SF & RF. So, both SF and RF are not met in indexer cluster. 

I tried to roll and resync bucket manually, that didn't work. There're no buckets in excess buckets, i've cleared them like more than 3hrs.

Is there any way to meet SF & RF without loosing data or bucket ? I even tried to restart Splunk process on that Indexer

Forgot to mention, i had a /opt/cold drive that has I/O error on an indexer. To get it fix i had stop Splunk and remove an indexer from indexer cluster, All other indexers are up and running since last night.  All 45 indexers in cluster-master are up and running and left it to bucket fixup tasks to fix and it also to rebalance overnight. When i check morning there're only 2 fixup tasks left one is in SF & one in RF. 

Does it also need manual data rebalance to perform from indexer-cluster as well ?

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

as you have had some I/O errors on your /opt/cold there is possibility that there are some buckets which are corrupted and cannot used anymore. You should find from _internal -log what cause that issue. Just search those buckets from it which you have on MC's view of SF&RF not met and in fixing task.

After you have identified those reasons you could decide how to proceed. Maybe just remove primary bucket and use your replicas or something else, but this is totally dependent on the reason what you found from internal.

What are your SF & RF and have you single site or multisite cluster?

Basically it should't need a data rebalancing unless your bucket count has totally unbalanced between indexers. You could see that e.g. via REST calls.

r. Ismo

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sandeepreddy947,

having an infrastructure like your (45 Indexers), the only thing is to open a ticket to Splunk Support.

ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...