Splunk Enterprise

index clustering does not distribute primary buckets well for indexes

schose
Builder

Hi forum,

I have a 2 peer single site (sf2, rf2) index cluster. We recognized that the primaries for indexes are not distributed even by using the search:

 

 

| rest splunk_server=local /services/cluster/master/buckets
| rex field=title "^(?<repl_index>[^\~]+)" | search repl_index="*" standalone=0 frozen=*
| rename title AS bucketID | fields bucketID peers.*.search_state peers.*.bucket_flags frozen repl_index
| rename peers.3DAB62DE-6D21-4C93-B8E5-A65370709B79.bucket_flags as bucketflags
| eval prim=if(bucketflags = "0x0","prim_yes","prim_no")
| stats count by repl_index prim
| xyseries repl_index prim count
| fillnull prim_yes,prim_no
| eval ratio=prim_yes/(prim_yes+prim_no)
| eval ratio=round(ratio*100,2)
| search repl_index="*"

 

 

 

sc-1.png

More or less all primaries are either on one indexer or the other, resulting in uneven load as we have a search hotspot on one index.

We were able to have a far better distribution after we set sf=1, removed excess buckets and set sf=2 again.

sc-2.png

Unfortunatly after stop an indexer for a while or do a rolling restart it's again very uneven distributed (as seen on the first screenshot).

it's also possible to get an even distribution when stopping clustermaster and peers at the same time and starting again - in this time we have data loss. restarting any component for it's own doesn't fix the issue.

we tried to rebalance primaries using:

 

 

curl -k -u admin:plaseentercreditcardnumber --request POST https://localhost:8089/services/cluster/master/control/control/rebalance_primaries

 

 

any hints how to fix this? We are using v8.0.7.

best regards,

Andreas

Labels (1)
Tags (1)
0 Karma

schose
Builder

Hi,

after further testing it looks like upgrading to v.8.2.1 fixes the issue. But I havn't found anything usefull in release notes. 😞

Regards,

Andreas

Tags (1)
0 Karma

schose
Builder

Hi all,

v8.1.5 also seems to have no issues.. 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...