Connection closes with INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed.

[mravindra]

TCP connection closes after few hours and will not re-establish even after splunk restart.
Connection gets re-established by editing outputs.conf and again closes with below logs after few hours.

05-23-2018 00:29:36.512 +0200 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
05-23-2018 00:29:36.512 +0200 INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed
05-23-2018 00:29:36.512 +0200 INFO TcpOutputProc - Will close stream to current indexer 10.x.x.x:9997
05-23-2018 00:29:36.512 +0200 INFO TcpOutputProc - Closing stream for idx=10.x.x.x:9997
05-23-2018 00:29:36.767 +0200 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
05-23-2018 00:30:06.567 +0200 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed

[renjith_nair]

Hi @mravindra,

If its between forwarder and indexer, please check the ssl configuration. looks like your server is not trusting the certificate presented by client. It could be because of various reasons including host name validation, common name mismatch etc.

http://docs.splunk.com/Documentation/Splunk/7.1.1/Security/ConfigureSplunkforwardingtousesignedcerti...

[SasiB137]

http://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Wheretofindtheconfigurationfiles
http://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4
http://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPra...

This is a whole cookbook intended to help you configure all the SSL things in Splunk correctly.

[SasiB137]

Right but the scenario here is "TCP connection closes after few hours and will not re-establish even after splunk restart." So not sure if it's a valid observation.

[SasiB137]

If we corrupt the forwarder certificate we should see the above logs getting emitted? Right ?