topic Re: Data cloning and load balancing in Getting Data In

Data cloning and load balancing

splunkreal — Tue, 29 Sep 2020 14:31:30 GMT

Hello guys,

we have this config for outputs.conf :

*[tcpout]
defaultGroup = ssl_splk_sitesAB_9997
useACK = true
maxQueueSize = 100MB

[tcpout:ssl_splk_sitesAB_9997]
server = s301lxidxxx01:9997, s302lxidxxx01:9997

[tcpout-server://s301lxidxxx01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=yyy
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS_SitesAB/local/cacert.2017.pem
sslVerifyServerCert=false

[tcpout-server://s302lxidxxx01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=yyy
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS_SitesAB/local/cacert.2017.pem
sslVerifyServerCert=false*

This is data cloning. However we would like to add a third server (s303...) only in case both first indexers fail. How would you do?

Thanks.

Re: Data cloning and load balancing

woodcock — Mon, 19 Jun 2017 14:09:01 GMT

This is generally done with index replication, which would be a more practical way to go (do not reinvent the wheel):

https://docs.splunk.com/Documentation/Splunk/6.6.0/Indexer/Aboutclusters

Re: Data cloning and load balancing

splunkreal — Mon, 19 Jun 2017 14:17:38 GMT

Hello woodcock, there is already index replication with s303 (third server) but it is not directly connected to forwarders. The current issue is if main indexers go down then agents won't send data until they are up again.

Re: Data cloning and load balancing

woodcock — Mon, 19 Jun 2017 14:21:56 GMT

Then you are doing your replication wrong. You setup an index cluster and then have the forwarder's outputs.conf settings controlled by the Cluster Master. The index replication in the indexer tier ensures that no data or searchability is lost if 1 more more indexers goes down. You are trying to manually implement a well-tested core feature and it is not a good idea. Read the document link that I posted.

Re: Data cloning and load balancing

splunkreal — Mon, 19 Jun 2017 14:49:48 GMT

Ok, is it the current implementation which is wrong or adding a third server as a loadbalancer? Thanks.

Re: Data cloning and load balancing

woodcock — Mon, 19 Jun 2017 14:54:46 GMT

Your implementation is wrong. Read more here:
http://docs.splunk.com/Documentation/Splunk/6.6.1/Indexer/indexerdiscovery

Re: Data cloning and load balancing

splunkreal — Mon, 19 Jun 2017 15:14:44 GMT

If I've understood, just one indexer should receive data then replicating automatically inside the index cluster? I think it has been implemented to have real time data on both indexers.

Re: Data cloning and load balancing

woodcock — Mon, 19 Jun 2017 15:20:43 GMT

Exactly. Then you have your forwarders auto-discover the AsOfRightNowWhichIndexersShouldBeUsedForRoundRobin instead of hard-coding a list.

Re: Data cloning and load balancing

sduchene_splunk — Mon, 19 Jun 2017 15:46:26 GMT

Regarding your point "this is data cloning" : this is not.
Your current configuration is load balancing. (a forwarder sends to indexers "round robinily" ) (Do i get additional points for making up words? 🙂
see example: https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Forwarding/Setuploadbalancingd

Then if indexer are in cluster, the data is replicated between indexers.

Regarding your question, Woodcock has answered ( and multisite cluster might be what you are looking for)

Re: Data cloning and load balancing

splunkreal — Tue, 20 Jun 2017 13:22:06 GMT

Ok, got it! We have only one target group so it's not data cloning. Thanks.