Getting Data In

Data cloning and load balancing

Builder

Hello guys,

we have this config for outputs.conf :

*[tcpout]
defaultGroup = ssl_splk_sitesAB_9997
useACK = true
maxQueueSize = 100MB

[tcpout:ssl_splk_sitesAB_9997]
server = s301lxidxxx01:9997, s302lxidxxx01:9997

[tcpout-server://s301lxidxxx01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=yyy
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS_SitesAB/local/cacert.2017.pem
sslVerifyServerCert=false

[tcpout-server://s302lxidxxx01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=yyy
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS_SitesAB/local/cacert.2017.pem
sslVerifyServerCert=false*

This is data cloning. However we would like to add a third server (s303...) only in case both first indexers fail. How would you do?

Thanks.

0 Karma
1 Solution

Splunk Employee
Splunk Employee

Regarding your point "this is data cloning" : this is not.
Your current configuration is load balancing. (a forwarder sends to indexers "round robinily" ) (Do i get additional points for making up words? 🙂
see example: https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Forwarding/Setuploadbalancingd

Then if indexer are in cluster, the data is replicated between indexers.

Regarding your question, Woodcock has answered ( and multisite cluster might be what you are looking for)

View solution in original post

Splunk Employee
Splunk Employee

Regarding your point "this is data cloning" : this is not.
Your current configuration is load balancing. (a forwarder sends to indexers "round robinily" ) (Do i get additional points for making up words? 🙂
see example: https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Forwarding/Setuploadbalancingd

Then if indexer are in cluster, the data is replicated between indexers.

Regarding your question, Woodcock has answered ( and multisite cluster might be what you are looking for)

View solution in original post

Builder

Ok, got it! We have only one target group so it's not data cloning. Thanks.

0 Karma

Esteemed Legend

This is generally done with index replication, which would be a more practical way to go (do not reinvent the wheel):

https://docs.splunk.com/Documentation/Splunk/6.6.0/Indexer/Aboutclusters

0 Karma

Builder

Hello woodcock, there is already index replication with s303 (third server) but it is not directly connected to forwarders. The current issue is if main indexers go down then agents won't send data until they are up again.

0 Karma

Esteemed Legend

Then you are doing your replication wrong. You setup an index cluster and then have the forwarder's outputs.conf settings controlled by the Cluster Master. The index replication in the indexer tier ensures that no data or searchability is lost if 1 more more indexers goes down. You are trying to manually implement a well-tested core feature and it is not a good idea. Read the document link that I posted.

0 Karma

Builder

If I've understood, just one indexer should receive data then replicating automatically inside the index cluster? I think it has been implemented to have real time data on both indexers.

0 Karma

Esteemed Legend

Exactly. Then you have your forwarders auto-discover the AsOfRightNowWhichIndexersShouldBeUsedForRoundRobin instead of hard-coding a list.

Builder

Ok, is it the current implementation which is wrong or adding a third server as a loadbalancer? Thanks.

0 Karma

Esteemed Legend
0 Karma