Deployment Architecture

Question ) How does Cluster Master decide Primary searchable copy ?

rbal_splunk
Splunk Employee
Splunk Employee

As you may know in Non-multi-site Cluster there is only one set of “primary” searchable buckets that respond to searches
With multi-site, each site can have searchable replicas that respond to searches- based on site affinity.
Clarify using the bucket rest end - how can we tell which bucket will be considered Primary searchable .

0 Karma

gjanders
SplunkTrust
SplunkTrust

After some discussion with Splunk support I wrote this search to determine how many primary buckets per indexer in site0:

| rest /services/cluster/master/buckets splunk_server=local 
| search `comment("This idea comes from Splunk support & https://answers.splunk.com/answers/234717/how-to-get-list-of-buckets-which-are-having-issues.html , attempt to determine the count of primary buckets per peer for site0. This report is designed to provide 1 example of a useful REST endpoint")` standalone=0 frozen=0
| rename primaries_by_site.site0 AS peerGUID
| join type=outer peerGUID [ rest /services/cluster/master/peers splunk_server=local
| fields active_* host* label title status site
| eval PeerName= site + ":" + label + ":" + host_port_pair
| rename title AS peerGUID
| rename site AS peerSite
| table peerGUID PeerName peerSite]
| stats count by PeerName
| chart sum(count) AS count by PeerName

Note that needs to run from either your cluster master or a node where the cluster master is a peer.

For the full details try something like:

| rest /services/cluster/master/buckets splunk_server=local 
| head 10

Feel free to run without the | head 10 but that may slowdown your browser 🙂
There are other primaries_by_site.* fields per-site FYI

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

In non muli-site clustering, its either 0x0, or 0xFFFFFFFF , basically primary or not primary.

The individual bits are only relevant in multi-site clustering.
The flags is a 64 bit bitmask, with the smallest bit corresponding to Primary for site0. The second smallest would be primary for site1, the third for site2, and so on....so 0x0 = primary for nothing (searches from any site will not get results for this bucket on this peer)

Bucket marked 0x1 = searches that come from searchheads with site=0 will get results! (primary for site0)
Bucket marked 0x2 = primary for site1, all site=site1 searches will get results from these buckets
Bucket marked 0x3 == (0x1+0x2) primary for site0 + site1, so searches from site0 SearchHeads and site1 searchheads will get results for this bucket.

On the cluster/master/buckets/BID endpoint, the masks should add up to whatever the mutl-site config is
for example, if We have available_sites=site2,site3, then the mask will need to have 0x1 (site0), 0x4(site2), 0x8 (site3) distributed among its indexers. in this example, if search_factor=1, then only 1 bucket will be searchable and should get assigned all the flags (0x13)

lguinn2
Legend

When I look at the bucket_flags using | rest /services/cluster/master/buckets, I see the following:

0x0
0x2
0x3
0x4
0x5
0x7

This is a multisite cluster with 2 sites; each site has 2 indexers. There have never been any more indexers or any other sites. The cluster is complete and valid; it is not in maintenance mode. What do these flags mean? I would expect to see 0x3 and 0x5 only...

I do have some nonclustered buckets (with flags of 0x7) and some single-site buckets (with a variety of flags)

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...