Why is heavy forwarder unable to send data to indexer?

[vksplunk1]

Hi Good Afternoon,

Our Heavy Forwarder is unable to forward to one of the indexer but able to send data another indexer. Here is what I saw in splunkd.log of Heavy Forwarder:

06-22-2022 13:24:03.471 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=xx.xx.xxx.xxx port=9996 _numberOfFailures=2
06-22-2022 13:24:03.473 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.473 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=yy.yy.yy.yy port=9996 _numberOfFailures=2

[richgalloway]

Was it ever able to connect?  If so, what changed since then?

Port 9996 is non-standard.  Is the HF using the correct port?  Are all indexers listening on that port?  Are all firewalls allowing connections to that port?

The quoted log is reporting errors connecting to two indexers rather than one.  Perhaps the problem is more widespread.

[vksplunk1]

Thank you for your response. If configure to send data to only one indexer it's working fine.  This is an issue only when HF outputs.conf to send data to multiple indexers based on a REGEX 

[richgalloway]

Please share the HF's outputs.conf (use btool).

[vksplunk1]

Thank you for looking in to. Here is OUTPUTS.CONF:

[tcpout:sndIndexers]
server = indexer1:9996
sslPassword = <<Password>>

[tcpout:tstIndexers]
server = indexer2:9996
sslPassword = <<password2>>

[vksplunk1]

Thank you for looking in to it. I furthur looked in to the logs and found errors from indexer(s) also:

On Heavy Forwarder:
====================
06-22-2022 13:24:03.471 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=xx.xx.xx.xx port=9996 _numberOfFailures=2
06-22-2022 13:24:03.473 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.473 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=yy.yy.yy.yy port=9996 _numberOfFailures=2

On Indexers:
==============

TcpInputProc [27164 FwdDataReceiverThread] - Error encountered for connection from src=xxxxx error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

[richgalloway]

That looks like an SSL problem.  Make sure the forwarder is using https and has the right certificate for the indexers.

[vksplunk1]

Thank you again.

Where can I find the password sslPassword for OUTPUTS.CONF

[tcpout:sndIndexers]
server = xx.xx.xx.xx:9996
sslPassword = 

[richgalloway]

Ask the person who created the certificate.  Failing that, you'll likely have to re-generate the certificate with a new password.

[richgalloway]

Where's the part where you send to multiple indexers using regex?

[vksplunk1]

It's in TRANSFORMS.CONF:

[xxxx]
REGEX=xxxxxx
DEST_KEY=_TCP_ROUTING
FORMAT=sndIndexers

[yyyy]
REGEX=yyyyy
DEST_KEY=_TCP_ROUTING
FORMAT=tstIndexers