ERROR TcpOutputFd [TcpOutEloop] - Read error. An existing connection was forcibly closed by the remote host.

[Balshore]

ERROR TcpOutputFd [TcpOutEloop] - Read error. An existing connection was forcibly closed by the remote host.

I have this error on one of our Splunk Windows UF, and I cannot resolve the issue.

our setup is UF>Intermediate HF>HF>Indexer

1. Connection from the UF to the intermediate HF port 9997 is okay, using tnc.

2. tried re installing the UF and restarting the Intermediate HF but still no success.

3. I noticed that from intermediate HF, its only sending SYN_RECV packets to UF.

Please help to check.
balshore

[Balshore]

will update you once rebooted the UF

[PickleRick]

The error itself means just what it says - the other end of the connection shut the connection before your end completed what it was doing (and maybe shut the connection on your end).

The actual cause might vary - it might be due to the other end getting "clogged" (probable if the connection is getting closed sometimes but generally works). It might be because of TLS mismatch (so your end initiates the connection but cannot complete the TLS negotiation).

You should look for more info in logs on both sides of the connection.

[Balshore]

Thanks for your reply.

I’ve been reviewing the splunkd.log, specifically searching for keywords such as “blocked”, “full”, “tcpin”, “error”, and “warn”. However, I haven’t been able to find any logs indicating errors related to the UF.

To provide more context, I have already successfully onboarded around 100 UFs, and this appears to be the only one encountering issues.

I also removed the inputs on the UF side to help isolate whether this could be an ingestion-related issue, but the problem persists.

Thanks,
Balshore

[PickleRick]

Does it happen in mid-connection or are you unable to establish any connectivity at all? Are you getting any logs from this UF?

If not, that would suggest either network problems or TLS mismatch (which is fairly unlikely if you have standardized config, to be honest).

If you're not getting any data at all, there is also a possibility of another quite nasty to debug issue - a IPS-like device in your network path. I've seen situations where an IPS (everyone had forgotten about) was detecting "incorrect" certificates and was sending TCP RST to both ends of the connection). That's an ugly one because on both ends it looks like the other end just closed the connection without any apparent reason. And on the network level you're getting a TCP RST packet from the other end and that's it.

[Balshore]

Thanks for the reply.

Actually we are not using any TLS or SSL connection from UF to Intermediate HF.

When I run the ./spunk list forward-server, the HF is inactive.

When I did the tcpdump, the HF is only sending SYN_RECV packets. Other UFs are established.

Last resort would be rebooting the UF servers already. (I already reinstall the UF agent)

Thanks,

Balshore

[PickleRick]

OK. That seems to suggest (don't know the whole picture) some network-related issues. Have you tried doing the tcpdump/wireshark on both ends of the connection and verifying whether the packets match?