As you may know in Non-multi-site Cluster there is only one set of “primary” searchable buckets that respond to searches
With multi-site, each site can have searchable replicas that respond to searches- based on site affinity.
Clarify using the bucket rest end - how can we tell which bucket will be considered Primary searchable .
In non muli-site clustering, its either 0x0, or 0xFFFFFFFF , basically primary or not primary.
The individual bits are only relevant in multi-site clustering.
The flags is a 64 bit bitmask, with the smallest bit corresponding to Primary for site0. The second smallest would be primary for site1, the third for site2, and so on....so 0x0 = primary for nothing (searches from any site will not get results for this bucket on this peer)
Bucket marked 0x1 = searches that come from searchheads with site=0 will get results! (primary for site0)
Bucket marked 0x2 = primary for site1, all site=site1 searches will get results from these buckets
Bucket marked 0x3 == (0x1+0x2) primary for site0 + site1, so searches from site0 SearchHeads and site1 searchheads will get results for this bucket.
On the cluster/master/buckets/BID endpoint, the masks should add up to whatever the mutl-site config is
for example, if We have availablesites=site2,site3, then the mask will need to have 0x1 (site0), 0x4(site2), 0x8 (site3) distributed among its indexers. in this example, if searchfactor=1, then only 1 bucket will be searchable and should get assigned all the flags (0x13)
When I look at the bucket_flags using
| rest /services/cluster/master/buckets, I see the following:
0x0 0x2 0x3 0x4 0x5 0x7
This is a multisite cluster with 2 sites; each site has 2 indexers. There have never been any more indexers or any other sites. The cluster is complete and valid; it is not in maintenance mode. What do these flags mean? I would expect to see 0x3 and 0x5 only...
I do have some nonclustered buckets (with flags of 0x7) and some single-site buckets (with a variety of flags)
After some discussion with Splunk support I wrote this search to determine how many primary buckets per indexer in site0:
| rest /services/cluster/master/buckets splunk_server=local | search `comment("This idea comes from Splunk support & https://answers.splunk.com/answers/234717/how-to-get-list-of-buckets-which-are-having-issues.html , attempt to determine the count of primary buckets per peer for site0. This report is designed to provide 1 example of a useful REST endpoint")` standalone=0 frozen=0 | rename primaries_by_site.site0 AS peerGUID | join type=outer peerGUID [ rest /services/cluster/master/peers splunk_server=local | fields active_* host* label title status site | eval PeerName= site + ":" + label + ":" + host_port_pair | rename title AS peerGUID | rename site AS peerSite | table peerGUID PeerName peerSite] | stats count by PeerName | chart sum(count) AS count by PeerName
Note that needs to run from either your cluster master or a node where the cluster master is a peer.
For the full details try something like:
| rest /services/cluster/master/buckets splunk_server=local | head 10
Feel free to run without the | head 10 but that may slowdown your browser 🙂
There are other primariesbysite.* fields per-site FYI