Getting Data In

Handle lost buckets

New Member

I have splunk cluster with sf=2, rf=2 (they are met).
It was maintained by another contractor, so I have no ideas about what caused the issue.
First, on master node for some indexes I see data copies marked grey and data copies number is 170/173, etc... same for serachable and replicated copies. And no bucket fixup tasks are running.
I guess this means that some buckets do not exist on both indexers, am I right?

Second, I ran dbinspect command for this index, it returned 173 results. tsidxState is "full" for every bucket.

How do I find problematic buckets and delete them to make Indexer clustering page green again?

UPD. I uploaded photo
alt text

0 Karma

Builder

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

Click Action for the bucket that you want to manage.
Select one of the available actions:
View bucket details
Roll
Resync
Delete Copy
A pop-up window appears to guide you through the selected action.
Use the following sequence when performing actions on anomalous bucket.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomal...

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

0 Karma

New Member

Thank you for your answer. As I mentioned before I have no buckets in fixup state when I click on gray ones. Now it looks like a visual bug but it still persists after server restart.

0 Karma

SplunkTrust
SplunkTrust

Have you tried to restart splunk on Cluster Master? Sometimes I have seen that CM flags indexes with grey color but on top of the screen it displays Search Factor and Replication Factor are met & after few minutes every indexes were green.

0 Karma

New Member

I tried, but it does not help. All blocks for particular indexes are constantly grey. You can check the photo I added to my question (can't make screenshot, sorry)

0 Karma

SplunkTrust
SplunkTrust

If you click on that grey icon for one of the index, it will take you to Bucket Status page, as you mentioned you don't have any bucket fixup running but can you please check any fixup tasks - pending ?

0 Karma

New Member

I was incorrect, actually I have no fixup tasks at all (pending or in progress), and no excess buckets btw

0 Karma

Builder

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

Click Action for the bucket that you want to manage.
Select one of the available actions:
View bucket details
Roll
Resync
Delete Copy
A pop-up window appears to guide you through the selected action.
Use the following sequence when performing actions on anomalous bucket.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomal...

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

0 Karma

Splunk Employee
Splunk Employee

Ivan, please post your method as an answer instead of a comment. It will help mark the question as answered.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!