Deployment Architecture

High Availability - Light Forwarders and combining cloning and autoLB

Glenn
Builder

We are looking to improve our current Splunk architecture (which is the same as the example in "Data cloning for high availability from http://www.splunk.com/wiki/Community:MultipleIndexServerDeploymentOptions, with one indexer each in two datacentres), to something capable of handling a higher volume, after we recently purchased larger Splunk licenses for the business.

This question is about the bottom end of the architecture, how the LightForwarders pass their data up the system to what we have already (provisionally) decided will be a “cluster” of indexers in each datacentre, being fed by an autoLB source. Searching will be done via a distributed search head in each DC, searching all of the "cluster" nodes in that same DC.

To do this, we need data to be cloned across both datacentre indexer clusters, but the data to be shared between cluster nodes via autoLB.

LightForwarders are required on the source servers, because of their small footprint. I want to know if this is possible for them.

outputs.conf:

[tcpout]
defaultGroup=dc1,dc2

[tcpout:dc1]
autoLB=true
autoLBFrequency=30
server=splunkindexers.dc1.company.com:42099

[tcpout:dc2]
autoLB=true
autoLBFrequency=30
server=splunkindexers.dc2.company.com:42099

Where splunkindexers.dc1.companyname.com is a DNS list that contains:
- indexer1.dc1.company.com
- indexer2.dc1.company.com

…and splunkindexers.dc2.companyname.com is a DNS list that contains:
- indexer3.dc2.company.com
- indexer4.dc2.company.com

This would combine the LightForwarder capability of cloning with the LightForwarder capability of autoLB to DNS lists, giving us what we want (cloning to two indexer clusters(one in each DC), but spreading the data between cluster nodes). Can LWFs handle the combination of its two capabilities at once?

Since there is no requirement to process any of the data (which is not possible with a LWF), and it only uses forwarding strategies that the LightForwarder is capable of (albeit a combination of them), I am hoping that it is… if it can do them separately why couldn’t it do both?

Cheers,

Glenn

1 Solution

jkerai
Splunk Employee
Splunk Employee

Yes, this configuration should work. You should be able to clone the data to 2 clusters of indexers for HA.

View solution in original post

jkerai
Splunk Employee
Splunk Employee

Yes, this configuration should work. You should be able to clone the data to 2 clusters of indexers for HA.

Glenn
Builder

Dang, now I find that it has already mostly been answered in http://answers.splunk.com/questions/421/is-it-possible-to-configure-cloning-and-autolb-simultaneousl... - and it looks like it is possible. Anyway, if anyone has a confirmation, or any useful comments for my new architecture proposale, it would be much appreciated.

0 Karma
Get Updates on the Splunk Community!

Testing out the OpenTelemetry Collector With raw Data

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

New Cloud Intrusion Detection System Add-on for Splunk

In July 2022 Splunk released the Cloud IDS add-on which expanded Splunk capabilities in security and data ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...