Getting Data In

Data cloning and load balancing

realsplunk
Builder

Hello guys,

we have this config for outputs.conf :

*[tcpout]
defaultGroup = ssl_splk_sitesAB_9997
useACK = true
maxQueueSize = 100MB

[tcpout:ssl_splk_sitesAB_9997]
server = s301lxidxxx01:9997, s302lxidxxx01:9997

[tcpout-server://s301lxidxxx01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=yyy
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS_SitesAB/local/cacert.2017.pem
sslVerifyServerCert=false

[tcpout-server://s302lxidxxx01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=yyy
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS_SitesAB/local/cacert.2017.pem
sslVerifyServerCert=false*

This is data cloning. However we would like to add a third server (s303...) only in case both first indexers fail. How would you do?

Thanks.

0 Karma
1 Solution

sduchene_splunk
Splunk Employee
Splunk Employee

Regarding your point "this is data cloning" : this is not.
Your current configuration is load balancing. (a forwarder sends to indexers "round robinily" ) (Do i get additional points for making up words? 🙂
see example: https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Forwarding/Setuploadbalancingd

Then if indexer are in cluster, the data is replicated between indexers.

Regarding your question, Woodcock has answered ( and multisite cluster might be what you are looking for)

View solution in original post

sduchene_splunk
Splunk Employee
Splunk Employee

Regarding your point "this is data cloning" : this is not.
Your current configuration is load balancing. (a forwarder sends to indexers "round robinily" ) (Do i get additional points for making up words? 🙂
see example: https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Forwarding/Setuploadbalancingd

Then if indexer are in cluster, the data is replicated between indexers.

Regarding your question, Woodcock has answered ( and multisite cluster might be what you are looking for)

View solution in original post

realsplunk
Builder

Ok, got it! We have only one target group so it's not data cloning. Thanks.

0 Karma

woodcock
Esteemed Legend

This is generally done with index replication, which would be a more practical way to go (do not reinvent the wheel):

https://docs.splunk.com/Documentation/Splunk/6.6.0/Indexer/Aboutclusters

0 Karma

realsplunk
Builder

Hello woodcock, there is already index replication with s303 (third server) but it is not directly connected to forwarders. The current issue is if main indexers go down then agents won't send data until they are up again.

0 Karma

woodcock
Esteemed Legend

Then you are doing your replication wrong. You setup an index cluster and then have the forwarder's outputs.conf settings controlled by the Cluster Master. The index replication in the indexer tier ensures that no data or searchability is lost if 1 more more indexers goes down. You are trying to manually implement a well-tested core feature and it is not a good idea. Read the document link that I posted.

0 Karma

realsplunk
Builder

If I've understood, just one indexer should receive data then replicating automatically inside the index cluster? I think it has been implemented to have real time data on both indexers.

0 Karma

woodcock
Esteemed Legend

Exactly. Then you have your forwarders auto-discover the AsOfRightNowWhichIndexersShouldBeUsedForRoundRobin instead of hard-coding a list.

realsplunk
Builder

Ok, is it the current implementation which is wrong or adding a third server as a loadbalancer? Thanks.

0 Karma

woodcock
Esteemed Legend
0 Karma