Do you know what the meaning of the following warning is?: "Bucket is not on any other peer! Removing it."

[D2SI]

Hello there,

We faced an issue with our Indexer Cluster and I am trying to understand what happened.

I see these messages :

07-25-2018 11:51:02.387 +0200 WARN CMMaster - event=removePeerBuckets peer= peer_name= bid= msg="Bucket is not on any other peer! Removing it."
07-25-2018 19:51:02.387 +0200 WARN CMMaster - event=removePeerBuckets peer= peer_name= bid= msg="Bucket is not on any other peer! Removing it."

It seems to be saying that the bucket has been removed, but I am still able to retrieve data from that particular bucket via a Splunk search and I can also spot both db and rb buckets on those IDX1 & IDX4 cold DBs.

Any idea ?

Thanks in advance,

[harsmarvania57 · Accepted solution]

Hi @D2SI,

These type of logs will generate when CM thinks that the Indexer is down (There are many possibilities for this for example: Indexer is down, network connectivity issue between CM and Indexer, Indexer too busy to respond to CM within stipulated time, CM too busy to respond to Indexer within stipulated time).

When CM and Indexer start communicating again, CM will add that bucket again in records. So, if you check logs properly, you will able to see logs that CM is again adding those buckets to Indexers

10-01-2018 00:04:49.391 +0100 INFO  CMMaster - Adding bid=main~123~8HDJRD1-A12B-123A-12AB-A123BC3D6767E (status='Complete' search_status='Searchable' mask=0 checksum= report_acc_summaries_size=0 data_model_summaries_size=0 standalone=no size=1709 genid=558 site=site1) to peer=8HDJRD1-A12B-123A-12AB-A123BC3D6767E peer_name=MYPEER

So based on my knowledge, this means those buckets are again searchable with new genid and the CM keeps records of it. However, these buckets didn't remove from Indexers and weren't added again to Indexers. It is just recording the update on CM with GenID and flags changing on those buckets (for example: Searchable to Searchable Pending Mask and Searchable Pending Mask to Searchable).

I guess this info will help you to understand what is going on in your environment.

[D2SI]

Alright so it is related to the CM records for searchable / non searchable buckets.

I indeed have that Adding bid message after some time.

Thanks for the explanation!

[harsmarvania57]

I have converted my comment to answer, if it really helps you then you can accept & upvote it.