Hi All,
I am trying to setup Splunk in multisite environment. I have two sites in environment. Details of my environment are:
Master Node : 1
Search Head : 2 ( Each in site)
Peers : 4 ( Site 1 : 3 and Site 2 : 1)
With my current configuration i am getting error "Indexing not ready, fewer than replication factor peers are up." My current configuration is as follows:
MasterNode
[general]
site=site1
[clustering]
available_sites=site1,site2
mode = master
multisite=true
pass4SymmKey = abcde
site_replication_factor = origin:2,total:3
site_search_factor = origin:1,total:2
Please validate and let me know correct configurations to be done on Master and peers. With the error, I guess my replication_factor does not meet the criteria. Any help would be appreciated.
you'll need to have at least X peers per site, where X is the site_replication_factor's 'origin' number.
in your case here, you'll need 2 peers per site before indexing starts.
you can skip this requirement by setting indexing-ready on the master:
./splunk set indexing-ready
but you're replication factor will never be met (since site2 origin buckets can't possibly have 2 copies!)
btw, its also recommended that you set 'replication_factor' to match X as well (not the site one, the older replication_factor). see http://docs.splunk.com/Documentation/Splunk/6.1/Indexer/Migratetomultisite#How_the_cluster_migrates_...
Hi @Sourabhv06
If @dxu_splunk's answer and comment helped solve your question, don't forget to accept their response by clicking on "Accept" right below their answer to resolve this post. Thanks!
Patrick
you'll need to have at least X peers per site, where X is the site_replication_factor's 'origin' number.
in your case here, you'll need 2 peers per site before indexing starts.
you can skip this requirement by setting indexing-ready on the master:
./splunk set indexing-ready
but you're replication factor will never be met (since site2 origin buckets can't possibly have 2 copies!)
btw, its also recommended that you set 'replication_factor' to match X as well (not the site one, the older replication_factor). see http://docs.splunk.com/Documentation/Splunk/6.1/Indexer/Migratetomultisite#How_the_cluster_migrates_...
I want 1 replication copy of data at each site i.e one copy at site 1 and 1 copy at site 2 irrespective of data origin. Is that possible if i have 3 indexers at site 1 and 1 indexer at site 2 ? Please suggest configuration for the same?
sure. set site_replication_factor=origin:1,total:2
Hi dux_Splunk,
Once i had set up site_replication_factor as origin:1,total:2, i am not able to start the masterNode Service. Seems error in configuration. Could you please help me in setting up my multisite cluster based on the configuration mentioned above. Appreciate your quick help.
Thanks,
Sourabh
you'll need to change "replication_factor" as well to be less than the 'total' site replication factor.. lets set it to 1. (this affects only buckets that were created when multisite was turned off)
Thanks!!! my replication factor and search factor are met after adding replication_factor and search_factor to my clustering stanza.
However i have a question. This search_factor and replication factor will be applicable for which bucktes ?
when u had multisite turned off, if there were buckets created during that time, they will follow replication_factor search_factor instead of site_replication_factor site_search_factor
My multisite cluster is configured fine but my data is not getting replicated. Under indexes i can only see _audit and _internal. I am not able tyo see _main. Request you to help ASAP
did you set your index to be repFactor=auto?
http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Configurethepeerindexes
Yes i had already distributed indexes.conf file with [main] stanza from master. I had mentioned repFactor = auto in that. But still no _main index is available.
Please suggest.
did u index data into main?
Thanks For pointing it out. I was checking it before forwarding Data..
For testing I created a index test at one of my peer and edited a indexes.conf with following details
[test]
repFactor = auto
Once i restarted the peers, At master i can see the test index but now i am getting error as "Replication fator is not met and Search Factor is not met".
Please suggest where i have gone wrong. My server.conf is same as you stated.
Moreover, while troubleshooting, I can see error for buckets as "cannot replicate as bucket is not servicable".
Appreciate your help!!!
whats ur rep factor, how many peers per site do u have? did you change the "replication_factor=1" and "search_factor=1"?
the "not serviceable" message goes away after a few minutes, dont worry too much about that and just check it again
I have two sites. SIte 1 has 3 peers and site 2 has 1 peer. yes i had changed replication_factor = 1 and search_factor = 1 . My current config is
site_replication_factor = origin:1,site1:1,site2:1,total:2
site_search_factor = origin:1,site1:1,site2:1,total:2
replication_factor = 1
search_factor = 1
I worked fine till i created a custom index. After indexing data to custom index , error as stated above is persisting.
Please suggest right config for my multisite cluster based on above information
can your peers communicate between each other? you'll need to make sure your peers' have their replication_port and mgmt_ports open to each other.
Yes i had enabled all ports and indexing under main index is working absolutely fine. Issue occurs when i create a custom index.
Once i create a custom index at one of my peer and than feed data to that index, at master i can see error as " replication factor not met " and " search factor not met" , though same config worked for main index
what are the warnings/errors in splunk_home/var/log/splunk/splunkd.log