topic Re: Connection closes with INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed. in Getting Data In

Connection closes with INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed.

mravindra — Tue, 29 Sep 2020 20:01:26 GMT

TCP connection closes after few hours and will not re-establish even after splunk restart.
Connection gets re-established by editing outputs.conf and again closes with below logs after few hours.

05-23-2018 00:29:36.512 +0200 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
05-23-2018 00:29:36.512 +0200 INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed
05-23-2018 00:29:36.512 +0200 INFO TcpOutputProc - Will close stream to current indexer 10.x.x.x:9997
05-23-2018 00:29:36.512 +0200 INFO TcpOutputProc - Closing stream for idx=10.x.x.x:9997
05-23-2018 00:29:36.767 +0200 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
05-23-2018 00:30:06.567 +0200 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed

Re: Connection closes with INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed.

renjith_nair — Wed, 20 Jun 2018 13:33:36 GMT

Hi @mravindra,

If its between forwarder and indexer, please check the ssl configuration. looks like your server is not trusting the certificate presented by client. It could be because of various reasons including host name validation, common name mismatch etc.

http://docs.splunk.com/Documentation/Splunk/7.1.1/Security/ConfigureSplunkforwardingtousesignedcertificates

Re: Connection closes with INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed.

SasiB137 — Wed, 20 Jun 2018 18:11:07 GMT

Right but the scenario here is "TCP connection closes after few hours and will not re-establish even after splunk restart." So not sure if it's a valid observation.

Re: Connection closes with INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed.

SasiB137 — Wed, 20 Jun 2018 18:17:58 GMT

If we corrupt the forwarder certificate we should see the above logs getting emitted? Right ?

Re: Connection closes with INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed.

SasiB137 — Thu, 21 Jun 2018 12:12:41 GMT

http://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Wheretofindtheconfigurationfiles
http://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4
http://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPractices.pdf

This is a whole cookbook intended to help you configure all the SSL things in Splunk correctly.