Deployment Architecture

error in clustering and replication

satyaallaparthi
Communicator

Hello,

I am getting the following error in my deployment server or Cluster master. Eventhough ouputs.conf is correct.

outputs.conf :

[tcpout]
defaultGroup = indexers
[tcpout:indexers]
server = IDX1:9997, IDX2:9997
autoLB = true
forceTimebasedAutoLB = true
autoLBFrequency = 40

Please help me with the error. Any help would be appreciated. Thanks!

TCPOutAutoLB-0
Root Cause(s):
More than 20% of forwarding destinations have failed. Ensure your hosts and ports in outputs.conf are correct. Also ensure that the indexers are all running, and that any SSL certificates being used for forwarding are correct.
Last 50 related messages:
05-12-2019 19:30:02.091 -0400 WARN TcpOutputProc - Cooked connection to ip=10.184.132.110:9997 timed out
05-12-2019 19:26:33.000 -0400 WARN TcpOutputProc - Cooked connection to ip=10.184.132.110:9997 timed out
05-12-2019 19:25:33.180 -0400 WARN TcpOutputProc - Cooked connection to ip=10.184.132.110:9997 timed out
05-12-2019 19:23:03.692 -0400 WARN TcpOutputProc - Cooked connection to ip=10.184.132.110:9997 timed out

0 Karma
1 Solution

DavidHourani
Super Champion

Hi @satyaallaparthi,

Seems like your configuration's working but the receiving indexers are either not listening on port 9997 or a firewall is blocking the traffic in between.

Have a look there and see. Easy way to debug that is to run telnet 10.184.132.110 9997 or nc 10.184.132.110 99997 from your forwarder to see if it times out or not.

Cheers,
David

View solution in original post

oscar84x
Contributor

Do you have any errors in splunkd.log?

0 Karma

DavidHourani
Super Champion

Hi @satyaallaparthi,

Seems like your configuration's working but the receiving indexers are either not listening on port 9997 or a firewall is blocking the traffic in between.

Have a look there and see. Easy way to debug that is to run telnet 10.184.132.110 9997 or nc 10.184.132.110 99997 from your forwarder to see if it times out or not.

Cheers,
David

satyaallaparthi
Communicator

Hello @DavidHourani,

I did tried telnet before I posted. Everything seems to be fine and communicating to each other. But still getting the problem.. :'(

Thanks,

0 Karma

koshyk
Super Champion

Before outputs.conf , you need to verify if the SSL/TLS connectivity is good with Indexer Slaves & CLM
I've made quite a new templates for clustering and can be found here (mainly I use for docker)

So sample in CLM should look like

[general]
site= site0

#indexer clustering
[clustering]
mode = master 
pass4SymmKey = my_pass
cluster_label = my_idx_cluster1
multisite = true
replication_factor = 2
search_factor = 2
site_replication_factor = origin:1, total:2
site_search_factor = origin:1, total:2
available_sites = site1, site2

[indexer_discovery]
pass4SymmKey = my_pass
polling_rate = 300

and Indexer Slaves like

#indexer clustering
[clustering]
mode = slave
master_uri=https://CLUSTER_MASTER_URI:8089
pass4SymmKey = my_pass
cluster_label = my_idx_cluster1

[replication_port://{{replicationPort}}]
disabled = false
rootCA = $SPLUNK_HOME/etc/apps/MY_CERT_APP/bin/auth/rootCA.pem
serverCert = $SPLUNK_HOME/etc/apps/MY_CERT_APP/bin/auth/device.pem
password = my_pass
requireClientCert = false

satyaallaparthi
Communicator

Hello @koshyk,

Everything looks exactly same in my server.conf and I did telnet from my CM to slaves and everything is communicating properly but still getting the above error in CM.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...