Handle lost buckets

[asnegina]

I have splunk cluster with sf=2, rf=2 (they are met).
It was maintained by another contractor, so I have no ideas about what caused the issue.
First, on master node for some indexes I see data copies marked grey and data copies number is 170/173, etc... same for serachable and replicated copies. And no bucket fixup tasks are running.
I guess this means that some buckets do not exist on both indexers, am I right?

Second, I ran dbinspect command for this index, it returned 173 results. tsidxState is "full" for every bucket.

How do I find problematic buckets and delete them to make Indexer clustering page green again?

UPD. I uploaded photo

[ivanreis]

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

Click Action for the bucket that you want to manage.
Select one of the available actions:
View bucket details
Roll
Resync
Delete Copy
A pop-up window appears to guide you through the selected action.
Use the following sequence when performing actions on anomalous bucket.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomal...

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

[asnegina]

Thank you for your answer. As I mentioned before I have no buckets in fixup state when I click on gray ones. Now it looks like a visual bug but it still persists after server restart.

[harsmarvania57]

Have you tried to restart splunk on Cluster Master? Sometimes I have seen that CM flags indexes with grey color but on top of the screen it displays Search Factor and Replication Factor are met & after few minutes every indexes were green.

[asnegina]

I tried, but it does not help. All blocks for particular indexes are constantly grey. You can check the photo I added to my question (can't make screenshot, sorry)

[harsmarvania57]

If you click on that grey icon for one of the index, it will take you to Bucket Status page, as you mentioned you don't have any bucket fixup running but can you please check any fixup tasks - pending ?

[asnegina]

I was incorrect, actually I have no fixup tasks at all (pending or in progress), and no excess buckets btw

[ivanreis]

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

Click Action for the bucket that you want to manage.
Select one of the available actions:
View bucket details
Roll
Resync
Delete Copy
A pop-up window appears to guide you through the selected action.
Use the following sequence when performing actions on anomalous bucket.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomal...

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

[yannK]

Ivan, please post your method as an answer instead of a comment. It will help mark the question as answered.