Deployment Architecture

Why are we getting "Replication factor not met" in our multisite indexer clustering environment?

Path Finder

We have configured multisite indexer clustering (2 peers at each site1/2 and one search head at site 1) with the below settings of server.conf of the master server and indexers .

Master server.conf

[general]
pass4SymmKey = $1$xNRfsRamx/pN
site = site1

[clustering]
available_sites = site1,site2
mode = master
multisite = true
pass4SymmKey = $1$9MxSqh+o6q08TJov
site_search_factor = origin:1,total:2
site_replication_factor = origin:2,total:3

.....................................
Indexers server.conf:

[general]
site = site1

[replication_port://7778]

[clustering]
master_uri = https://x.x.x.x:8089
mode = slave
pass4SymmKey = whatever

We are getting “Replication factor not met” error on the master server's dashboard and "Missing enough suitable candidates to create replicated copy in order to meet replication policy. Missing={ site2:1 } " error.

Only 4 audit and 4 _internal index buckets are not replicating. All the rest and main index buckets are replicating ok .plz help

0 Karma
1 Solution

Splunk Employee
Splunk Employee

These are likely pre-multisite buckets. (you can tell by going to the cluster master endpoint /services/cluster/master/buckets?filter=replicationcount<3, and note that the buckets there have constraintooriginsite = 1)

see answers question / docs

you can try setting replicationfactor=2 in the cluster master server.conf, and restart the master - this should then show rf/sf met. (we don't replicate pre multisite buckets across sites, so since you have 2 indexers per site, setting replicationfactor=2 makes sense)

View solution in original post

Splunk Employee
Splunk Employee

These are likely pre-multisite buckets. (you can tell by going to the cluster master endpoint /services/cluster/master/buckets?filter=replicationcount<3, and note that the buckets there have constraintooriginsite = 1)

see answers question / docs

you can try setting replicationfactor=2 in the cluster master server.conf, and restart the master - this should then show rf/sf met. (we don't replicate pre multisite buckets across sites, so since you have 2 indexers per site, setting replicationfactor=2 makes sense)

View solution in original post

Path Finder

thanks dxusplunk ...Even I also feel that these are the pre multisite clustering buckets .. The replication of main index does not give any error however 4 buckets each of audit /internal index are not replicating .

As you responded we should keep replication factor=2 .But in that case if any disaster happens at site 1(say site 1 goes down) and if all the replicated buckets are only residing at site 1 since there are only two replicated buckets (site replication factor=2) then how would be the disaster recovery can happen ?

your help would be appreciated ..

0 Karma

Splunk Employee
Splunk Employee

replicationfactor only affects non multisite buckets (those 4+4 buckets you mentioned). we do not replicate them across sites (if the source bucket is on siteA, it'll be replicated within siteA), so they do not have site disaster recovery. for your actual data that came in after multisite, those buckets/data follow sitereplicationfactor/sitesearch_factor, so all should be well.

Path Finder

It worked 🙂 🙂

Thanks a Lot dxusplunk & esixsplunk ..

0 Karma

Splunk Employee
Splunk Employee

You will need to keep the replicationfactor and searchfactor configurations under the clustering option. This is legacy support for local, non-clustered, indexes before multisite is configured.
It will have no effect on multisite replicated buckets, as these use the site_* configurations.

This is a known issue.

Communicator

Hi Darshan
Try rolling hot buckets by running following command

splunk internal call /data/indexes/audit/roll-hot-buckets -auth admin:changeme

Wait for some time and see if problem gets resolved

Path Finder

nopes ..same state ...

0 Karma