topic Re: Heavy Forwarder Unable to send data to Indexer in Splunk Enterprise

Why is heavy forwarder unable to send data to indexer?

vksplunk1 — Thu, 23 Jun 2022 18:54:43 GMT

Hi Good Afternoon,

Our Heavy Forwarder is unable to forward to one of the indexer but able to send data another indexer. Here is what I saw in splunkd.log of Heavy Forwarder:

06-22-2022 13:24:03.471 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=xx.xx.xxx.xxx port=9996 _numberOfFailures=2
06-22-2022 13:24:03.473 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.473 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=yy.yy.yy.yy port=9996 _numberOfFailures=2

Re: Heavy Forwarder Unable to send data to Indexer

richgalloway — Wed, 22 Jun 2022 19:06:41 GMT

Was it ever able to connect? If so, what changed since then?

Port 9996 is non-standard. Is the HF using the correct port? Are all indexers listening on that port? Are all firewalls allowing connections to that port?

The quoted log is reporting errors connecting to two indexers rather than one. Perhaps the problem is more widespread.

Re: Heavy Forwarder Unable to send data to Indexer

vksplunk1 — Wed, 22 Jun 2022 20:08:22 GMT

Thank you for your response. If configure to send data to only one indexer it's working fine. This is an issue only when HF outputs.conf to send data to multiple indexers based on a REGEX

Re: Heavy Forwarder Unable to send data to Indexer

richgalloway — Wed, 22 Jun 2022 23:59:30 GMT

Please share the HF's outputs.conf (use btool).

Re: Heavy Forwarder Unable to send data to Indexer

vksplunk1 — Thu, 23 Jun 2022 19:16:53 GMT

Thank you for looking in to. Here is OUTPUTS.CONF:

[tcpout:sndIndexers]
server = indexer1:9996
sslPassword = <<Password>>

[tcpout:tstIndexers]
server = indexer2:9996
sslPassword = <<password2>>

Re: Heavy Forwarder Unable to send data to Indexer

richgalloway — Thu, 23 Jun 2022 19:36:32 GMT

Where's the part where you send to multiple indexers using regex?

Re: Heavy Forwarder Unable to send data to Indexer

vksplunk1 — Thu, 23 Jun 2022 20:36:10 GMT

It's in TRANSFORMS.CONF:

[xxxx]
REGEX=xxxxxx
DEST_KEY=_TCP_ROUTING
FORMAT=sndIndexers

[yyyy]
REGEX=yyyyy
DEST_KEY=_TCP_ROUTING
FORMAT=tstIndexers

Re: Heavy Forwarder Unable to send data to Indexer

vksplunk1 — Fri, 24 Jun 2022 13:25:20 GMT

Thank you for looking in to it. I furthur looked in to the logs and found errors from indexer(s) also:

On Heavy Forwarder:
====================
06-22-2022 13:24:03.471 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=xx.xx.xx.xx port=9996 _numberOfFailures=2
06-22-2022 13:24:03.473 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.473 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=yy.yy.yy.yy port=9996 _numberOfFailures=2

On Indexers:
==============

TcpInputProc [27164 FwdDataReceiverThread] - Error encountered for connection from src=xxxxx error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Re: Heavy Forwarder Unable to send data to Indexer

richgalloway — Fri, 24 Jun 2022 14:15:15 GMT

That looks like an SSL problem. Make sure the forwarder is using https and has the right certificate for the indexers.

Re: Heavy Forwarder Unable to send data to Indexer

vksplunk1 — Fri, 24 Jun 2022 19:00:03 GMT

Thank you again.

Where can I find the password sslPassword for OUTPUTS.CONF

[tcpout:sndIndexers]
server = xx.xx.xx.xx:9996
sslPassword =

Re: Heavy Forwarder Unable to send data to Indexer

richgalloway — Fri, 24 Jun 2022 20:14:36 GMT

Ask the person who created the certificate. Failing that, you'll likely have to re-generate the certificate with a new password.