topic Re: Question ) How does Cluster Master decide Primary searchable copy ? in Deployment Architecture

Question ) How does Cluster Master decide Primary searchable copy ?

rbal_splunk — Thu, 19 Oct 2017 06:27:32 GMT

As you may know in Non-multi-site Cluster there is only one set of “primary” searchable buckets that respond to searches
With multi-site, each site can have searchable replicas that respond to searches- based on site affinity.
Clarify using the bucket rest end - how can we tell which bucket will be considered Primary searchable .

Re: Question ) How does Cluster Master decide Primary searchable copy ?

rbal_splunk — Tue, 29 Sep 2020 16:17:07 GMT

In non muli-site clustering, its either 0x0, or 0xFFFFFFFF , basically primary or not primary.

The individual bits are only relevant in multi-site clustering.
The flags is a 64 bit bitmask, with the smallest bit corresponding to Primary for site0. The second smallest would be primary for site1, the third for site2, and so on....so 0x0 = primary for nothing (searches from any site will not get results for this bucket on this peer)

Bucket marked 0x1 = searches that come from searchheads with site=0 will get results! (primary for site0)
Bucket marked 0x2 = primary for site1, all site=site1 searches will get results from these buckets
Bucket marked 0x3 == (0x1+0x2) primary for site0 + site1, so searches from site0 SearchHeads and site1 searchheads will get results for this bucket.

On the cluster/master/buckets/BID endpoint, the masks should add up to whatever the mutl-site config is
for example, if We have available_sites=site2,site3, then the mask will need to have 0x1 (site0), 0x4(site2), 0x8 (site3) distributed among its indexers. in this example, if search_factor=1, then only 1 bucket will be searchable and should get assigned all the flags (0x13)

Re: Question ) How does Cluster Master decide Primary searchable copy ?

lguinn2 — Tue, 13 Feb 2018 23:38:54 GMT

When I look at the bucket_flags using | rest /services/cluster/master/buckets, I see the following:

0x0
0x2
0x3
0x4
0x5
0x7

This is a multisite cluster with 2 sites; each site has 2 indexers. There have never been any more indexers or any other sites. The cluster is complete and valid; it is not in maintenance mode. What do these flags mean? I would expect to see 0x3 and 0x5 only...

I do have some nonclustered buckets (with flags of 0x7) and some single-site buckets (with a variety of flags)

Re: Question ) How does Cluster Master decide Primary searchable copy ?

gjanders — Tue, 29 Sep 2020 18:07:52 GMT

After some discussion with Splunk support I wrote this search to determine how many primary buckets per indexer in site0:

| rest /services/cluster/master/buckets splunk_server=local 
| search `comment("This idea comes from Splunk support & https://answers.splunk.com/answers/234717/how-to-get-list-of-buckets-which-are-having-issues.html , attempt to determine the count of primary buckets per peer for site0. This report is designed to provide 1 example of a useful REST endpoint")` standalone=0 frozen=0
| rename primaries_by_site.site0 AS peerGUID
| join type=outer peerGUID [ rest /services/cluster/master/peers splunk_server=local
| fields active_* host* label title status site
| eval PeerName= site + ":" + label + ":" + host_port_pair
| rename title AS peerGUID
| rename site AS peerSite
| table peerGUID PeerName peerSite]
| stats count by PeerName
| chart sum(count) AS count by PeerName

Note that needs to run from either your cluster master or a node where the cluster master is a peer.

For the full details try something like:

| rest /services/cluster/master/buckets splunk_server=local 
| head 10

Feel free to run without the | head 10 but that may slowdown your browser 🙂
There are other primaries_by_site.* fields per-site FYI