Getting Data In
Highlighted

How to get list of buckets which are having issues in replicating, from API and CLI?

Explorer

When my splunk multi-site indexer cluster comes up, I have some buckets belonging to _audit and _internal which are having issues getting replicated, due to which Indexer clustering dashboard on Cluster Master shows, Replication Factor not met. I can see the bucket names from the dashboard page, by clicking on the bucket status button. Then when I delete those buckets from Cluster Master CLI, everything is back to normal and my dashboard says "Rep. factor met".

I want to know instead of Splunk dashboard UI, is there a way to get the bucket names which are having replication issues, via CLI or REST API?

Highlighted

Re: How to get list of buckets which are having issues in replicating, from API and CLI?

Splunk Employee
Splunk Employee

This search comes courtesy of my co-worker @Masa

Clustering

Multi-site enabled

Simple version of bucket state by site

| rest /services/cluster/master/buckets
   | rex field=title "^(?[^\~]+)"
   | search repl_index="*" standalone=0 frozen=0
   | rename title AS bucketID
   | fields bucketID  *origin_site* *_by_site*
   | untable bucketID siteState value
   | rex mode=sed field=siteState "s/\./__/"
   | rex mode=sed field=siteState "s/_count_/_/"
   | search NOT siteState=primaries_*
   | xyseries bucketID siteState value
   | fields - search_by_site
   | fillnull
   | eval rep_total= rep_by_site__site1 + rep_by_site__site2 + rep_by_site__site3
   | eval srch_total = search_by_site__site1 + search_by_site__site2 + search_by_site__site3
   | rename constrain_to_origin_site AS constrain
   | rename origin_site AS origin
   | rename rep_by_site__site1 AS rep_site1
   | rename rep_by_site__site2 AS rep_site2
   | rename rep_by_site__site3 AS rep_site3
   | rename search_by_site__site1 AS srch_site1
   | rename search_by_site__site2 AS srch_site2
   | rename search_by_site__site3 AS srch_site3

table output:

bucketID constrain origin repsite1 repsite2 repsite3 reptotal srchsite1 srchsite2 srchsite3 srchtotal


_audit~118~FF782A13-8AFB-4617-BCB4-15ED11928DD7 0 site1 2 1 1 4 2 1 1 4
_audit~119~FF782A13-8AFB-4617-BCB4-15ED11928DD7 0 site1 2 1 2 5 2 1 1 4

You can further filter out for buckets where rep or search factor is not met (assuming your rep factor=4 and search factor=3) by appending this to the end of the search:
| search reptotal<4 OR srchtotal<3

Note: remove references to site3 in the search if you only have 2 sites in the multi-site cluster

Clustering

Multi-site enabled

| rest /services/cluster/master/buckets
   | rex field=title "^(?[^\~]+)"
   | search repl_index="*" standalone=0 frozen=0
   | rename title AS bucketID
   | fields bucketID peers.*.search_state  *site*
   | untable bucketID siteState value

   | rex field=siteState "peers\.(?[^\.]*?)\.(?search_state)"
   | rex field=siteState "(?primaries_by_site)\.(?\S+)"
   | rex field=siteState "(?rep_count_by_site)\.(?\S+)"
   | rex field=siteState "(?search_count_by_site)\.(?\S+)"

   | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
   | eval site=if(siteState=="origin_site", value, site)
   | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
   | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)

   | join type=outer peerGUID [ rest /services/cluster/master/peers
                          | fields active_* host* label title status site
                          | eval PeerName= site + ":" + label + ":" + host_port_pair
                          | rename title AS peerGUID
                          | rename site AS peerSite
                          | table peerGUID PeerName peerSite ]
   | eval site=if(siteState=="search_state", peerSite, site)
   | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value)
   | eval value=if(siteState=="search_state", PeerName + ":" + value, value)
   | fields - PeerName peerGUID peerSite    | chart values(value) over bucketID by siteState

table output:

               bucketID                     constrain origin                     primaries_by_site                      rep_by_site

srchbysite search_state



audit~118~FF782A13-8AFB-4617-BCB4-15ED11928DD7 0 site1 site1:centos58-64sup01-620CP:10.140.48.137:55591:Forsite1 site1:2 site1:2

site1:centos58-64sup01-620CP:10.140.48.137:55591:Searchable
site2:centos65-64sup14-620CP:10.140.48.150:55591:Forsite2 site2:1 site2:1

site1:centos65-64sup06-620CP:10.140.48.142:55591:Searchable
site3:centos62-64sup13-620CP:10.140.48.149:55591:For
site3 site3:1 site3:1

site2:centos65-64sup14-620CP:10.140.48.150:55591:Searchable

site3:centos62-64sup13-620CP:10.140.48.149:55591:Searchable

audit~119~FF782A13-8AFB-4617-BCB4-15ED11928DD7 0 site1 site1:centos58-64sup01-620CP:10.140.48.137:55591:Forsite1 site1:2 site1:2

site1:centos58-64sup01-620CP:10.140.48.137:55591:Searchable
site2:centos65-64sup14-620CP:10.140.48.150:55591:Forsite2 site2:1 site2:1

site1:centos65-64sup06-620CP:10.140.48.142:55591:Searchable
site3:centos62-64sup13-620CP:10.140.48.149:55591:For
site3 site3:2 site3:1

site2:centos65-64sup14-620CP:10.140.48.150:55591:Searchable

site3:centos62-64sup12-620CP:10.140.48.148:55591:Unsearchable

site3:centos62-64sup13-620CP:10.140.48.149:55591:Searchable

Highlighted

Re: How to get list of buckets which are having issues in replicating, from API and CLI?

Splunk Employee
Splunk Employee

I think there was a problem with copying the regex extraction. The original search should be

| rest /services/cluster/master/buckets splunk_server=*
    | rex field=title "^(?<repl_index>[^\~]+)" 
    | search repl_index="*" standalone=0 frozen=0
    | rename title AS bucketID
    | fields bucketID peers.*.search_state  *site*
    | untable bucketID siteState value
    | rex field=siteState "peers\.(?<peerGUID>[^\.]*)\.(search_state)"
    | rex field=siteState "(?<siteState>primaries_by_site)\.(\S+)"
    | rex field=siteState "(?<siteState>rep_count_by_site)\.(\S+)"
    | rex field=siteState "(?<siteState>search_count_by_site)\.(\S+)"| eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
    | eval site=if(siteState=="origin_site", value, site)
    | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
    | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)

       | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
       | eval site=if(siteState=="origin_site", value, site)
       | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
       | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)

    | join type=outer peerGUID [ rest /services/cluster/master/peers splunk_server=*
                           | fields active_* host* label title status site
                           | eval PeerName= site + ":" + label + ":" + host_port_pair
                           | rename title AS peerGUID
                           | rename site AS peerSite
                           | table peerGUID PeerName peerSite ]
    | eval site=if(siteState=="search_state", peerSite, site)
    | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value)
    | eval value=if(siteState=="search_state", PeerName + ":" + value, value)
    | fields - PeerName peerGUID peerSite    | chart values(value) over bucketID by siteState
0 Karma