Splunk Enterprise

Why is heavy forwarder unable to send data to indexer?

vksplunk1
Explorer

Hi Good Afternoon,

Our Heavy Forwarder is unable to forward to one of the indexer but able to send data another indexer. Here is what I saw in splunkd.log of Heavy Forwarder:

06-22-2022 13:24:03.471 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=xx.xx.xxx.xxx port=9996 _numberOfFailures=2
06-22-2022 13:24:03.473 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.473 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=yy.yy.yy.yy port=9996 _numberOfFailures=2

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Was it ever able to connect?  If so, what changed since then?

Port 9996 is non-standard.  Is the HF using the correct port?  Are all indexers listening on that port?  Are all firewalls allowing connections to that port?

The quoted log is reporting errors connecting to two indexers rather than one.  Perhaps the problem is more widespread.

---
If this reply helps you, Karma would be appreciated.
0 Karma

vksplunk1
Explorer

Thank you for your response. If configure to send data to only one indexer it's working fine.  This is an issue only when HF outputs.conf to send data to multiple indexers based on a REGEX 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please share the HF's outputs.conf (use btool).

---
If this reply helps you, Karma would be appreciated.
0 Karma

vksplunk1
Explorer

Thank you for looking in to. Here is OUTPUTS.CONF:

 

[tcpout:sndIndexers]
server = indexer1:9996
sslPassword = <<Password>>

[tcpout:tstIndexers]
server = indexer2:9996
sslPassword = <<password2>>

0 Karma

vksplunk1
Explorer

Thank you for looking in to it. I furthur looked in to the logs and found errors from indexer(s) also:

 

On Heavy Forwarder:
====================
06-22-2022 13:24:03.471 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.472 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=xx.xx.xx.xx port=9996 _numberOfFailures=2
06-22-2022 13:24:03.473 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer
06-22-2022 13:24:03.473 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=yy.yy.yy.yy port=9996 _numberOfFailures=2

On Indexers:
==============

TcpInputProc [27164 FwdDataReceiverThread] - Error encountered for connection from src=xxxxx error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That looks like an SSL problem.  Make sure the forwarder is using https and has the right certificate for the indexers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

vksplunk1
Explorer

Thank you again.

 

Where can I find the password sslPassword for OUTPUTS.CONF

 

[tcpout:sndIndexers]
server = xx.xx.xx.xx:9996
sslPassword = 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Ask the person who created the certificate.  Failing that, you'll likely have to re-generate the certificate with a new password.

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Where's the part where you send to multiple indexers using regex?

---
If this reply helps you, Karma would be appreciated.
0 Karma

vksplunk1
Explorer

It's in TRANSFORMS.CONF:

[xxxx]
REGEX=xxxxxx
DEST_KEY=_TCP_ROUTING
FORMAT=sndIndexers

[yyyy]
REGEX=yyyyy
DEST_KEY=_TCP_ROUTING
FORMAT=tstIndexers

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...