Splunk Enterprise

index clustering does not distribute primary buckets well for indexes

schose
Builder

Hi forum,

I have a 2 peer single site (sf2, rf2) index cluster. We recognized that the primaries for indexes are not distributed even by using the search:

 

 

| rest splunk_server=local /services/cluster/master/buckets
| rex field=title "^(?<repl_index>[^\~]+)" | search repl_index="*" standalone=0 frozen=*
| rename title AS bucketID | fields bucketID peers.*.search_state peers.*.bucket_flags frozen repl_index
| rename peers.3DAB62DE-6D21-4C93-B8E5-A65370709B79.bucket_flags as bucketflags
| eval prim=if(bucketflags = "0x0","prim_yes","prim_no")
| stats count by repl_index prim
| xyseries repl_index prim count
| fillnull prim_yes,prim_no
| eval ratio=prim_yes/(prim_yes+prim_no)
| eval ratio=round(ratio*100,2)
| search repl_index="*"

 

 

 

sc-1.png

More or less all primaries are either on one indexer or the other, resulting in uneven load as we have a search hotspot on one index.

We were able to have a far better distribution after we set sf=1, removed excess buckets and set sf=2 again.

sc-2.png

Unfortunatly after stop an indexer for a while or do a rolling restart it's again very uneven distributed (as seen on the first screenshot).

it's also possible to get an even distribution when stopping clustermaster and peers at the same time and starting again - in this time we have data loss. restarting any component for it's own doesn't fix the issue.

we tried to rebalance primaries using:

 

 

curl -k -u admin:plaseentercreditcardnumber --request POST https://localhost:8089/services/cluster/master/control/control/rebalance_primaries

 

 

any hints how to fix this? We are using v8.0.7.

best regards,

Andreas

Labels (1)
Tags (1)
0 Karma

schose
Builder

Hi,

after further testing it looks like upgrading to v.8.2.1 fixes the issue. But I havn't found anything usefull in release notes. 😞

Regards,

Andreas

Tags (1)
0 Karma

schose
Builder

Hi all,

v8.1.5 also seems to have no issues.. 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...