Deployment Architecture

High Availability - Light Forwarders and combining cloning and autoLB

Builder

We are looking to improve our current Splunk architecture (which is the same as the example in "Data cloning for high availability from http://www.splunk.com/wiki/Community:MultipleIndexServerDeploymentOptions, with one indexer each in two datacentres), to something capable of handling a higher volume, after we recently purchased larger Splunk licenses for the business.

This question is about the bottom end of the architecture, how the LightForwarders pass their data up the system to what we have already (provisionally) decided will be a “cluster” of indexers in each datacentre, being fed by an autoLB source. Searching will be done via a distributed search head in each DC, searching all of the "cluster" nodes in that same DC.

To do this, we need data to be cloned across both datacentre indexer clusters, but the data to be shared between cluster nodes via autoLB.

LightForwarders are required on the source servers, because of their small footprint. I want to know if this is possible for them.

outputs.conf:

[tcpout]
defaultGroup=dc1,dc2

[tcpout:dc1]
autoLB=true
autoLBFrequency=30
server=splunkindexers.dc1.company.com:42099

[tcpout:dc2]
autoLB=true
autoLBFrequency=30
server=splunkindexers.dc2.company.com:42099

Where splunkindexers.dc1.companyname.com is a DNS list that contains:
- indexer1.dc1.company.com
- indexer2.dc1.company.com

…and splunkindexers.dc2.companyname.com is a DNS list that contains:
- indexer3.dc2.company.com
- indexer4.dc2.company.com

This would combine the LightForwarder capability of cloning with the LightForwarder capability of autoLB to DNS lists, giving us what we want (cloning to two indexer clusters(one in each DC), but spreading the data between cluster nodes). Can LWFs handle the combination of its two capabilities at once?

Since there is no requirement to process any of the data (which is not possible with a LWF), and it only uses forwarding strategies that the LightForwarder is capable of (albeit a combination of them), I am hoping that it is… if it can do them separately why couldn’t it do both?

Cheers,

Glenn

1 Solution

Splunk Employee
Splunk Employee

Yes, this configuration should work. You should be able to clone the data to 2 clusters of indexers for HA.

View solution in original post

Splunk Employee
Splunk Employee

Yes, this configuration should work. You should be able to clone the data to 2 clusters of indexers for HA.

View solution in original post

Builder

Dang, now I find that it has already mostly been answered in http://answers.splunk.com/questions/421/is-it-possible-to-configure-cloning-and-autolb-simultaneousl... - and it looks like it is possible. Anyway, if anyone has a confirmation, or any useful comments for my new architecture proposale, it would be much appreciated.

0 Karma