Data cloning and load balancing

[realsplunk]

Hello guys,

we have this config for outputs.conf :

*[tcpout]
defaultGroup = ssl_splk_sitesAB_9997
useACK = true
maxQueueSize = 100MB

[tcpout:ssl_splk_sitesAB_9997]
server = s301lxidxxx01:9997, s302lxidxxx01:9997

[tcpout-server://s301lxidxxx01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=yyy
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS_SitesAB/local/cacert.2017.pem
sslVerifyServerCert=false

[tcpout-server://s302lxidxxx01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=yyy
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS_SitesAB/local/cacert.2017.pem
sslVerifyServerCert=false*

This is data cloning. However we would like to add a third server (s303...) only in case both first indexers fail. How would you do?

Thanks.

[sduchene_splunk · Accepted solution]

Regarding your point "this is data cloning" : this is not.
Your current configuration is load balancing. (a forwarder sends to indexers "round robinily" ) (Do i get additional points for making up words? 🙂
see example: https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Forwarding/Setuploadbalancingd

Then if indexer are in cluster, the data is replicated between indexers.

Regarding your question, Woodcock has answered ( and multisite cluster might be what you are looking for)

[realsplunk]

Ok, got it! We have only one target group so it's not data cloning. Thanks.

[woodcock]

This is generally done with index replication, which would be a more practical way to go (do not reinvent the wheel):

https://docs.splunk.com/Documentation/Splunk/6.6.0/Indexer/Aboutclusters

[realsplunk]

Hello woodcock, there is already index replication with s303 (third server) but it is not directly connected to forwarders. The current issue is if main indexers go down then agents won't send data until they are up again.

[woodcock]

Then you are doing your replication wrong. You setup an index cluster and then have the forwarder's outputs.conf settings controlled by the Cluster Master. The index replication in the indexer tier ensures that no data or searchability is lost if 1 more more indexers goes down. You are trying to manually implement a well-tested core feature and it is not a good idea. Read the document link that I posted.

[realsplunk]

If I've understood, just one indexer should receive data then replicating automatically inside the index cluster? I think it has been implemented to have real time data on both indexers.

[woodcock]

Exactly. Then you have your forwarders auto-discover the AsOfRightNowWhichIndexersShouldBeUsedForRoundRobin instead of hard-coding a list.

[realsplunk]

Ok, is it the current implementation which is wrong or adding a third server as a loadbalancer? Thanks.

[woodcock]

Your implementation is wrong. Read more here:
http://docs.splunk.com/Documentation/Splunk/6.6.1/Indexer/indexerdiscovery