Forwarder to Splunk cloud suddenly stop: error tcpoutputfd 1158951 connection to host 9997 failed ssl error = no error

[ThuLe]

Hello,

I have  HF and UF act as intermediate forwarders and forward logs to Splunk Cloud. We installed the credentials (.spl file) download from Splunk Cloud, and the forwarders were working fine until November 28, they have stopped sending the log to Splunk cloud. The error in splunkd.log is:

Network team confirm that they not change anything on the FW

We have searched and tried re-downloading the credentials (.spl file) from Splunk Cloud and reinstalled them on the forwarders, but the same errors persist. The errors only disappear when we disable the credentials app 100_<cloud instance>_splunkcloud.

Has anyone experienced this issue? Please help

Thank you very much

[PrewinThomas]

@ThuLe 

Most probably network related.
-Check connectivity to Splunk Cloud -

telnet inputs1.STACKID.splunkcloud.com 9997

-Check Firewall/SSL inspection recently enabled or changed

#https://splunk.my.site.com/customer/s/article/Splunk-Universal-Forwarder-is-not-sending-events-to-Sp...

Regards,
Prewin
🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[livehybrid]

Hi @ThuLe 

From my experience with 104 socket error (ECONNRESET) it has been a firewall issue every time, even when I was told there was no firewall between the two points (spoiler - there was!) 

I found that where a firewall was either blocking the content or attempting SSL introspection then it can cause the 104 error (which is ECONNRESET)

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

As far as I remember, error 104 means problems on a tcp connection level. Troubleshoot the connection with your typical network-level tools (tcpdump, netcat...) and verify your firewall config/logs.