topic Handle lost buckets in Getting Data In

Handle lost buckets

asnegina — Wed, 30 Oct 2019 08:09:57 GMT

I have splunk cluster with sf=2, rf=2 (they are met).
It was maintained by another contractor, so I have no ideas about what caused the issue.
First, on master node for some indexes I see data copies marked grey and data copies number is 170/173, etc... same for serachable and replicated copies. And no bucket fixup tasks are running.
I guess this means that some buckets do not exist on both indexers, am I right?

Second, I ran dbinspect command for this index, it returned 173 results. tsidxState is "full" for every bucket.

How do I find problematic buckets and delete them to make Indexer clustering page green again?

UPD. I uploaded photo

Re: Handle lost buckets

harsmarvania57 — Wed, 30 Oct 2019 09:27:22 GMT

Have you tried to restart splunk on Cluster Master? Sometimes I have seen that CM flags indexes with grey color but on top of the screen it displays Search Factor and Replication Factor are met & after few minutes every indexes were green.

Re: Handle lost buckets

asnegina — Wed, 30 Oct 2019 10:04:40 GMT

I tried, but it does not help. All blocks for particular indexes are constantly grey. You can check the photo I added to my question (can't make screenshot, sorry)

Re: Handle lost buckets

harsmarvania57 — Wed, 30 Oct 2019 11:12:36 GMT

If you click on that grey icon for one of the index, it will take you to Bucket Status page, as you mentioned you don't have any bucket fixup running but can you please check any fixup tasks - pending ?

Re: Handle lost buckets

asnegina — Wed, 30 Oct 2019 11:56:32 GMT

I was incorrect, actually I have no fixup tasks at all (pending or in progress), and no excess buckets btw

Re: Handle lost buckets

ivanreis — Mon, 04 Nov 2019 06:29:35 GMT

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

Click Action for the bucket that you want to manage.
Select one of the available actions:
View bucket details
Roll
Resync
Delete Copy
A pop-up window appears to guide you through the selected action.
Use the following sequence when performing actions on anomalous bucket.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomalous_bucket

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

Re: Handle lost buckets

yannK — Wed, 13 Nov 2019 21:53:10 GMT

Ivan, please post your method as an answer instead of a comment. It will help mark the question as answered.

Re: Handle lost buckets

ivanreis — Wed, 13 Nov 2019 23:56:12 GMT

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomalous_bucket

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

Re: Handle lost buckets

asnegina — Thu, 14 Nov 2019 15:38:29 GMT

Thank you for your answer. As I mentioned before I have no buckets in fixup state when I click on gray ones. Now it looks like a visual bug but it still persists after server restart.