Deployment Architecture

Why am I getting error "Indexing not ready, fewer than replication factor peers are up" with my current Splunk 6.1 multisite cluster configuration?

Sourabhv05
Communicator

Hi All,

I am trying to setup Splunk in multisite environment. I have two sites in environment. Details of my environment are:

Master Node : 1
Search Head : 2 ( Each in site)
Peers : 4 ( Site 1 : 3 and Site 2 : 1)

With my current configuration i am getting error "Indexing not ready, fewer than replication factor peers are up." My current configuration is as follows:

MasterNode

[general]
site=site1

[clustering]

available_sites=site1,site2
mode = master
multisite=true
pass4SymmKey = abcde
site_replication_factor = origin:2,total:3
site_search_factor = origin:1,total:2

Please validate and let me know correct configurations to be done on Master and peers. With the error, I guess my replication_factor does not meet the criteria. Any help would be appreciated.

1 Solution

dxu_splunk
Splunk Employee
Splunk Employee

you'll need to have at least X peers per site, where X is the site_replication_factor's 'origin' number.

in your case here, you'll need 2 peers per site before indexing starts.

you can skip this requirement by setting indexing-ready on the master:

./splunk set indexing-ready

but you're replication factor will never be met (since site2 origin buckets can't possibly have 2 copies!)

btw, its also recommended that you set 'replication_factor' to match X as well (not the site one, the older replication_factor). see http://docs.splunk.com/Documentation/Splunk/6.1/Indexer/Migratetomultisite#How_the_cluster_migrates_...

View solution in original post

ppablo
Retired

Hi @Sourabhv06

If @dxu_splunk's answer and comment helped solve your question, don't forget to accept their response by clicking on "Accept" right below their answer to resolve this post. Thanks!

Patrick

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

you'll need to have at least X peers per site, where X is the site_replication_factor's 'origin' number.

in your case here, you'll need 2 peers per site before indexing starts.

you can skip this requirement by setting indexing-ready on the master:

./splunk set indexing-ready

but you're replication factor will never be met (since site2 origin buckets can't possibly have 2 copies!)

btw, its also recommended that you set 'replication_factor' to match X as well (not the site one, the older replication_factor). see http://docs.splunk.com/Documentation/Splunk/6.1/Indexer/Migratetomultisite#How_the_cluster_migrates_...

Sourabhv05
Communicator

I want 1 replication copy of data at each site i.e one copy at site 1 and 1 copy at site 2 irrespective of data origin. Is that possible if i have 3 indexers at site 1 and 1 indexer at site 2 ? Please suggest configuration for the same?

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

sure. set site_replication_factor=origin:1,total:2

Sourabhv05
Communicator

Hi dux_Splunk,

Once i had set up site_replication_factor as origin:1,total:2, i am not able to start the masterNode Service. Seems error in configuration. Could you please help me in setting up my multisite cluster based on the configuration mentioned above. Appreciate your quick help.

Thanks,
Sourabh

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

you'll need to change "replication_factor" as well to be less than the 'total' site replication factor.. lets set it to 1. (this affects only buckets that were created when multisite was turned off)

0 Karma

Sourabhv05
Communicator

Thanks!!! my replication factor and search factor are met after adding replication_factor and search_factor to my clustering stanza.

However i have a question. This search_factor and replication factor will be applicable for which bucktes ?

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

see http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Migratetomultisite#How_the_cluster_migrate...

when u had multisite turned off, if there were buckets created during that time, they will follow replication_factor search_factor instead of site_replication_factor site_search_factor

0 Karma

Sourabhv05
Communicator

My multisite cluster is configured fine but my data is not getting replicated. Under indexes i can only see _audit and _internal. I am not able tyo see _main. Request you to help ASAP

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee
0 Karma

Sourabhv05
Communicator

Yes i had already distributed indexes.conf file with [main] stanza from master. I had mentioned repFactor = auto in that. But still no _main index is available.

Please suggest.

dxu_splunk
Splunk Employee
Splunk Employee

did u index data into main?

0 Karma

Sourabhv05
Communicator

Thanks For pointing it out. I was checking it before forwarding Data..

For testing I created a index test at one of my peer and edited a indexes.conf with following details

[test]
repFactor = auto

Once i restarted the peers, At master i can see the test index but now i am getting error as "Replication fator is not met and Search Factor is not met".

Please suggest where i have gone wrong. My server.conf is same as you stated.

0 Karma

Sourabhv05
Communicator

Moreover, while troubleshooting, I can see error for buckets as "cannot replicate as bucket is not servicable".

Appreciate your help!!!

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

whats ur rep factor, how many peers per site do u have? did you change the "replication_factor=1" and "search_factor=1"?

the "not serviceable" message goes away after a few minutes, dont worry too much about that and just check it again

0 Karma

Sourabhv05
Communicator

I have two sites. SIte 1 has 3 peers and site 2 has 1 peer. yes i had changed replication_factor = 1 and search_factor = 1 . My current config is

site_replication_factor = origin:1,site1:1,site2:1,total:2
site_search_factor = origin:1,site1:1,site2:1,total:2
replication_factor = 1
search_factor = 1

I worked fine till i created a custom index. After indexing data to custom index , error as stated above is persisting.

0 Karma

Sourabhv05
Communicator

Please suggest right config for my multisite cluster based on above information

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

can your peers communicate between each other? you'll need to make sure your peers' have their replication_port and mgmt_ports open to each other.

0 Karma

Sourabhv05
Communicator

Yes i had enabled all ports and indexing under main index is working absolutely fine. Issue occurs when i create a custom index.

Once i create a custom index at one of my peer and than feed data to that index, at master i can see error as " replication factor not met " and " search factor not met" , though same config worked for main index

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

what are the warnings/errors in splunk_home/var/log/splunk/splunkd.log

0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...