Splunk forwarder connection failure, even though the ports are opened.

[nadlurinadluri]

HI Team, I have opened the ports to the heavyforwarder from the UF. The ports are opened from UF to HF, but not the other way around. Getting the below message, when receiving the data.
curl: (56) Failure when receiving data from the peer

./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
xx.x.xxx.xxx:8201

Messages in output of splunkd.log
Indexer uri xx.x.xxx.xxx:8201, client refCount=0, client=NULL
DEBUG TcpOutputProc - Found quarantined idx=xx.x.xxx.xxx:8201 . skipping over it.
DEBUG TcpOutputProc - Looking for indexer...
DEBUG TcpOutputProc - Cannot find any valid descriptors when looking for new indexer.
DEBUG TcpOutputProc - Connection not available. Waiting for connection ...
DEBUG TcpOutputProc - channel not registered yet

Can you please suggest , what might be the issue that I am not able to connect to the HF?

Also, should we have a two way connection b/w the UF and HF?

[diogofgm]

validate that the connection is open. On the forwarder side do this

./splunk cmd openssl s_client -connect <ip>:<port> -showcerts

if the connection is good you should see connected and if you're using ssl you'll the HF cert details

[nadlurinadluri]

The connection is successful got the below when ran the query you have provided on the connection part.
CONNECTED(00000003)
write:errno=104

[diogofgm]

8201 is your receiving port in the indexer?
can you post your inputs conf from the indexer?

/opt/splunk/bin/splunk btool inputs list --debug splunktcp | grep -v system/default

[nadlurinadluri]

are you asking the inputs.conf from the indexer(which is HF in this case) or UF? why is that?

And yes the receiving port is 8201

[splunktcp://8201]
disabled=0

[diogofgm]

you should get something like this for a non ssl port:

CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 176 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1566926525
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

[nadlurinadluri]

yes, you are absolutely right, this is what I got as output.

[solarboyz1]

write:errno=104

On Linux, 104 is ECONNRESET for "Connection reset by peer" – in other words, the connection was forcibly closed with a TCP RST packet, either sent out by the server or spoofed by an intermediary.

[p_gurav]

try restarting UF once. The HF configured but not active.

[nadlurinadluri]

I did restart it good number of times, but no luck.

[p_gurav]

Could you share the inputs.conf file on HF?