Deployment Architecture

Do you know what the meaning of the following warning is?: "Bucket is not on any other peer! Removing it."

D2SI
Communicator

Hello there,

We faced an issue with our Indexer Cluster and I am trying to understand what happened.

I see these messages :

07-25-2018 11:51:02.387 +0200 WARN CMMaster - event=removePeerBuckets peer= peer_name= bid= msg="Bucket is not on any other peer! Removing it."
07-25-2018 19:51:02.387 +0200 WARN CMMaster - event=removePeerBuckets peer= peer_name= bid= msg="Bucket is not on any other peer! Removing it."

It seems to be saying that the bucket has been removed, but I am still able to retrieve data from that particular bucket via a Splunk search and I can also spot both db and rb buckets on those IDX1 & IDX4 cold DBs.

Any idea ?

Thanks in advance,

1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi @D2SI,

These type of logs will generate when CM thinks that the Indexer is down (There are many possibilities for this for example: Indexer is down, network connectivity issue between CM and Indexer, Indexer too busy to respond to CM within stipulated time, CM too busy to respond to Indexer within stipulated time).

When CM and Indexer start communicating again, CM will add that bucket again in records. So, if you check logs properly, you will able to see logs that CM is again adding those buckets to Indexers

10-01-2018 00:04:49.391 +0100 INFO  CMMaster - Adding bid=main~123~8HDJRD1-A12B-123A-12AB-A123BC3D6767E (status='Complete' search_status='Searchable' mask=0 checksum= report_acc_summaries_size=0 data_model_summaries_size=0 standalone=no size=1709 genid=558 site=site1) to peer=8HDJRD1-A12B-123A-12AB-A123BC3D6767E peer_name=MYPEER

So based on my knowledge, this means those buckets are again searchable with new genid and the CM keeps records of it. However, these buckets didn't remove from Indexers and weren't added again to Indexers. It is just recording the update on CM with GenID and flags changing on those buckets (for example: Searchable to Searchable Pending Mask and Searchable Pending Mask to Searchable).

I guess this info will help you to understand what is going on in your environment.

View solution in original post

harsmarvania57
SplunkTrust
SplunkTrust

Hi @D2SI,

These type of logs will generate when CM thinks that the Indexer is down (There are many possibilities for this for example: Indexer is down, network connectivity issue between CM and Indexer, Indexer too busy to respond to CM within stipulated time, CM too busy to respond to Indexer within stipulated time).

When CM and Indexer start communicating again, CM will add that bucket again in records. So, if you check logs properly, you will able to see logs that CM is again adding those buckets to Indexers

10-01-2018 00:04:49.391 +0100 INFO  CMMaster - Adding bid=main~123~8HDJRD1-A12B-123A-12AB-A123BC3D6767E (status='Complete' search_status='Searchable' mask=0 checksum= report_acc_summaries_size=0 data_model_summaries_size=0 standalone=no size=1709 genid=558 site=site1) to peer=8HDJRD1-A12B-123A-12AB-A123BC3D6767E peer_name=MYPEER

So based on my knowledge, this means those buckets are again searchable with new genid and the CM keeps records of it. However, these buckets didn't remove from Indexers and weren't added again to Indexers. It is just recording the update on CM with GenID and flags changing on those buckets (for example: Searchable to Searchable Pending Mask and Searchable Pending Mask to Searchable).

I guess this info will help you to understand what is going on in your environment.

D2SI
Communicator

Alright so it is related to the CM records for searchable / non searchable buckets.

I indeed have that Adding bid message after some time.

Thanks for the explanation!

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

I have converted my comment to answer, if it really helps you then you can accept & upvote it.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...