Splunk Enterprise

index clustering does not distribute primary buckets well for indexes

schose
Contributor

Hi forum,

I have a 2 peer single site (sf2, rf2) index cluster. We recognized that the primaries for indexes are not distributed even by using the search:

 

 

| rest splunk_server=local /services/cluster/master/buckets
| rex field=title "^(?<repl_index>[^\~]+)" | search repl_index="*" standalone=0 frozen=*
| rename title AS bucketID | fields bucketID peers.*.search_state peers.*.bucket_flags frozen repl_index
| rename peers.3DAB62DE-6D21-4C93-B8E5-A65370709B79.bucket_flags as bucketflags
| eval prim=if(bucketflags = "0x0","prim_yes","prim_no")
| stats count by repl_index prim
| xyseries repl_index prim count
| fillnull prim_yes,prim_no
| eval ratio=prim_yes/(prim_yes+prim_no)
| eval ratio=round(ratio*100,2)
| search repl_index="*"

 

 

 

sc-1.png

More or less all primaries are either on one indexer or the other, resulting in uneven load as we have a search hotspot on one index.

We were able to have a far better distribution after we set sf=1, removed excess buckets and set sf=2 again.

sc-2.png

Unfortunatly after stop an indexer for a while or do a rolling restart it's again very uneven distributed (as seen on the first screenshot).

it's also possible to get an even distribution when stopping clustermaster and peers at the same time and starting again - in this time we have data loss. restarting any component for it's own doesn't fix the issue.

we tried to rebalance primaries using:

 

 

curl -k -u admin:plaseentercreditcardnumber --request POST https://localhost:8089/services/cluster/master/control/control/rebalance_primaries

 

 

any hints how to fix this? We are using v8.0.7.

best regards,

Andreas

Labels (1)
Tags (1)
0 Karma

schose
Contributor

Hi,

after further testing it looks like upgrading to v.8.2.1 fixes the issue. But I havn't found anything usefull in release notes. 😞

Regards,

Andreas

Tags (1)
0 Karma

schose
Contributor

Hi all,

v8.1.5 also seems to have no issues.. 

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!