Getting Data In

Handle lost buckets

asnegina
New Member

I have splunk cluster with sf=2, rf=2 (they are met).
It was maintained by another contractor, so I have no ideas about what caused the issue.
First, on master node for some indexes I see data copies marked grey and data copies number is 170/173, etc... same for serachable and replicated copies. And no bucket fixup tasks are running.
I guess this means that some buckets do not exist on both indexers, am I right?

Second, I ran dbinspect command for this index, it returned 173 results. tsidxState is "full" for every bucket.

How do I find problematic buckets and delete them to make Indexer clustering page green again?

UPD. I uploaded photo
alt text

0 Karma

ivanreis
Builder

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

Click Action for the bucket that you want to manage.
Select one of the available actions:
View bucket details
Roll
Resync
Delete Copy
A pop-up window appears to guide you through the selected action.
Use the following sequence when performing actions on anomalous bucket.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomal...

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

0 Karma

asnegina
New Member

Thank you for your answer. As I mentioned before I have no buckets in fixup state when I click on gray ones. Now it looks like a visual bug but it still persists after server restart.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Have you tried to restart splunk on Cluster Master? Sometimes I have seen that CM flags indexes with grey color but on top of the screen it displays Search Factor and Replication Factor are met & after few minutes every indexes were green.

asnegina
New Member

I tried, but it does not help. All blocks for particular indexes are constantly grey. You can check the photo I added to my question (can't make screenshot, sorry)

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

If you click on that grey icon for one of the index, it will take you to Bucket Status page, as you mentioned you don't have any bucket fixup running but can you please check any fixup tasks - pending ?

0 Karma

asnegina
New Member

I was incorrect, actually I have no fixup tasks at all (pending or in progress), and no excess buckets btw

0 Karma

ivanreis
Builder

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

Click Action for the bucket that you want to manage.
Select one of the available actions:
View bucket details
Roll
Resync
Delete Copy
A pop-up window appears to guide you through the selected action.
Use the following sequence when performing actions on anomalous bucket.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomal...

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

0 Karma

yannK
Splunk Employee
Splunk Employee

Ivan, please post your method as an answer instead of a comment. It will help mark the question as answered.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...