Deployment Architecture

Why am I getting error "Indexing not ready, fewer than replication factor peers are up" with my current Splunk 6.1 multisite cluster configuration?

Communicator

Hi All,

I am trying to setup Splunk in multisite environment. I have two sites in environment. Details of my environment are:

Master Node : 1
Search Head : 2 ( Each in site)
Peers : 4 ( Site 1 : 3 and Site 2 : 1)

With my current configuration i am getting error "Indexing not ready, fewer than replication factor peers are up." My current configuration is as follows:

MasterNode

[general]
site=site1

[clustering]

available_sites=site1,site2
mode = master
multisite=true
pass4SymmKey = abcde
site_replication_factor = origin:2,total:3
site_search_factor = origin:1,total:2

Please validate and let me know correct configurations to be done on Master and peers. With the error, I guess my replication_factor does not meet the criteria. Any help would be appreciated.

1 Solution

Splunk Employee
Splunk Employee

you'll need to have at least X peers per site, where X is the site_replication_factor's 'origin' number.

in your case here, you'll need 2 peers per site before indexing starts.

you can skip this requirement by setting indexing-ready on the master:

./splunk set indexing-ready

but you're replication factor will never be met (since site2 origin buckets can't possibly have 2 copies!)

btw, its also recommended that you set 'replication_factor' to match X as well (not the site one, the older replication_factor). see http://docs.splunk.com/Documentation/Splunk/6.1/Indexer/Migratetomultisite#How_the_cluster_migrates_...

View solution in original post

Community Manager
Community Manager

Hi @Sourabhv06

If @dxu_splunk's answer and comment helped solve your question, don't forget to accept their response by clicking on "Accept" right below their answer to resolve this post. Thanks!

Patrick

0 Karma

Splunk Employee
Splunk Employee

you'll need to have at least X peers per site, where X is the site_replication_factor's 'origin' number.

in your case here, you'll need 2 peers per site before indexing starts.

you can skip this requirement by setting indexing-ready on the master:

./splunk set indexing-ready

but you're replication factor will never be met (since site2 origin buckets can't possibly have 2 copies!)

btw, its also recommended that you set 'replication_factor' to match X as well (not the site one, the older replication_factor). see http://docs.splunk.com/Documentation/Splunk/6.1/Indexer/Migratetomultisite#How_the_cluster_migrates_...

View solution in original post

Communicator

I want 1 replication copy of data at each site i.e one copy at site 1 and 1 copy at site 2 irrespective of data origin. Is that possible if i have 3 indexers at site 1 and 1 indexer at site 2 ? Please suggest configuration for the same?

0 Karma

Splunk Employee
Splunk Employee

sure. set site_replication_factor=origin:1,total:2

Communicator

Hi dux_Splunk,

Once i had set up site_replication_factor as origin:1,total:2, i am not able to start the masterNode Service. Seems error in configuration. Could you please help me in setting up my multisite cluster based on the configuration mentioned above. Appreciate your quick help.

Thanks,
Sourabh

0 Karma

Splunk Employee
Splunk Employee

you'll need to change "replication_factor" as well to be less than the 'total' site replication factor.. lets set it to 1. (this affects only buckets that were created when multisite was turned off)

0 Karma

Communicator

Thanks!!! my replication factor and search factor are met after adding replication_factor and search_factor to my clustering stanza.

However i have a question. This search_factor and replication factor will be applicable for which bucktes ?

0 Karma

Splunk Employee
Splunk Employee

see http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Migratetomultisite#How_the_cluster_migrate...

when u had multisite turned off, if there were buckets created during that time, they will follow replication_factor search_factor instead of site_replication_factor site_search_factor

0 Karma

Communicator

My multisite cluster is configured fine but my data is not getting replicated. Under indexes i can only see _audit and _internal. I am not able tyo see _main. Request you to help ASAP

0 Karma

Splunk Employee
Splunk Employee
0 Karma

Communicator

Yes i had already distributed indexes.conf file with [main] stanza from master. I had mentioned repFactor = auto in that. But still no _main index is available.

Please suggest.

Splunk Employee
Splunk Employee

did u index data into main?

0 Karma

Communicator

Thanks For pointing it out. I was checking it before forwarding Data..

For testing I created a index test at one of my peer and edited a indexes.conf with following details

[test]
repFactor = auto

Once i restarted the peers, At master i can see the test index but now i am getting error as "Replication fator is not met and Search Factor is not met".

Please suggest where i have gone wrong. My server.conf is same as you stated.

0 Karma

Communicator

Moreover, while troubleshooting, I can see error for buckets as "cannot replicate as bucket is not servicable".

Appreciate your help!!!

0 Karma

Splunk Employee
Splunk Employee

whats ur rep factor, how many peers per site do u have? did you change the "replication_factor=1" and "search_factor=1"?

the "not serviceable" message goes away after a few minutes, dont worry too much about that and just check it again

0 Karma

Communicator

I have two sites. SIte 1 has 3 peers and site 2 has 1 peer. yes i had changed replication_factor = 1 and search_factor = 1 . My current config is

site_replication_factor = origin:1,site1:1,site2:1,total:2
site_search_factor = origin:1,site1:1,site2:1,total:2
replication_factor = 1
search_factor = 1

I worked fine till i created a custom index. After indexing data to custom index , error as stated above is persisting.

0 Karma

Communicator

Please suggest right config for my multisite cluster based on above information

0 Karma

Splunk Employee
Splunk Employee

can your peers communicate between each other? you'll need to make sure your peers' have their replication_port and mgmt_ports open to each other.

0 Karma

Communicator

Yes i had enabled all ports and indexing under main index is working absolutely fine. Issue occurs when i create a custom index.

Once i create a custom index at one of my peer and than feed data to that index, at master i can see error as " replication factor not met " and " search factor not met" , though same config worked for main index

0 Karma

Splunk Employee
Splunk Employee

what are the warnings/errors in splunk_home/var/log/splunk/splunkd.log

0 Karma