Universal forwarder reporting "Error in SSL_read = 110" when forwarding to separate indexers.

[matthewssa]

Hi,

Is anyone familiar with this error code? I thought it may be the SSL certificates or a connection based issue but..

1. Checked cert expiration dates
2. Checked port 9998 came up and was using SSL in Splunk logs.
3. Checked on the syslogs to connect to both sets of indexers via OpenSSL and both return good connections. OpenSSL > s_client -connect x.x.x.x:9998
4. Saw connection attempts on tcpdump from the syslog ip addresses.

5. One set of indexers with the same certificates but behind a different firewall works. While another set of indexers show these errors and are not sending logs during the times we see these errors.

   06-06-2019 23:31:39.538 +0000 INFO TcpOutputProc - Connection to x.x.x.x:xxxx closed. default Error in SSL_read = 110, SSL Error = error : 00000000:lib(0):func(0):reason(0)