Deployment Architecture

error in clustering and replication

satyaallaparthi
Path Finder

Hello,

I am getting the following error in my deployment server or Cluster master. Eventhough ouputs.conf is correct.

outputs.conf :

[tcpout]
defaultGroup = indexers
[tcpout:indexers]
server = IDX1:9997, IDX2:9997
autoLB = true
forceTimebasedAutoLB = true
autoLBFrequency = 40

Please help me with the error. Any help would be appreciated. Thanks!

TCPOutAutoLB-0
Root Cause(s):
More than 20% of forwarding destinations have failed. Ensure your hosts and ports in outputs.conf are correct. Also ensure that the indexers are all running, and that any SSL certificates being used for forwarding are correct.
Last 50 related messages:
05-12-2019 19:30:02.091 -0400 WARN TcpOutputProc - Cooked connection to ip=10.184.132.110:9997 timed out
05-12-2019 19:26:33.000 -0400 WARN TcpOutputProc - Cooked connection to ip=10.184.132.110:9997 timed out
05-12-2019 19:25:33.180 -0400 WARN TcpOutputProc - Cooked connection to ip=10.184.132.110:9997 timed out
05-12-2019 19:23:03.692 -0400 WARN TcpOutputProc - Cooked connection to ip=10.184.132.110:9997 timed out

0 Karma
1 Solution

DavidHourani
Super Champion

Hi @satyaallaparthi,

Seems like your configuration's working but the receiving indexers are either not listening on port 9997 or a firewall is blocking the traffic in between.

Have a look there and see. Easy way to debug that is to run telnet 10.184.132.110 9997 or nc 10.184.132.110 99997 from your forwarder to see if it times out or not.

Cheers,
David

View solution in original post

0 Karma

oscar84x
Contributor

Do you have any errors in splunkd.log?

0 Karma

DavidHourani
Super Champion

Hi @satyaallaparthi,

Seems like your configuration's working but the receiving indexers are either not listening on port 9997 or a firewall is blocking the traffic in between.

Have a look there and see. Easy way to debug that is to run telnet 10.184.132.110 9997 or nc 10.184.132.110 99997 from your forwarder to see if it times out or not.

Cheers,
David

View solution in original post

0 Karma

satyaallaparthi
Path Finder

Hello @DavidHourani,

I did tried telnet before I posted. Everything seems to be fine and communicating to each other. But still getting the problem.. :'(

Thanks,

0 Karma

koshyk
Super Champion

Before outputs.conf , you need to verify if the SSL/TLS connectivity is good with Indexer Slaves & CLM
I've made quite a new templates for clustering and can be found here (mainly I use for docker)

So sample in CLM should look like

[general]
site= site0

#indexer clustering
[clustering]
mode = master 
pass4SymmKey = my_pass
cluster_label = my_idx_cluster1
multisite = true
replication_factor = 2
search_factor = 2
site_replication_factor = origin:1, total:2
site_search_factor = origin:1, total:2
available_sites = site1, site2

[indexer_discovery]
pass4SymmKey = my_pass
polling_rate = 300

and Indexer Slaves like

#indexer clustering
[clustering]
mode = slave
master_uri=https://CLUSTER_MASTER_URI:8089
pass4SymmKey = my_pass
cluster_label = my_idx_cluster1

[replication_port://{{replicationPort}}]
disabled = false
rootCA = $SPLUNK_HOME/etc/apps/MY_CERT_APP/bin/auth/rootCA.pem
serverCert = $SPLUNK_HOME/etc/apps/MY_CERT_APP/bin/auth/device.pem
password = my_pass
requireClientCert = false
0 Karma

satyaallaparthi
Path Finder

Hello @koshyk,

Everything looks exactly same in my server.conf and I did telnet from my CM to slaves and everything is communicating properly but still getting the above error in CM.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!