Getting Data In

Connection closes with INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed.

mravindra
Engager

TCP connection closes after few hours and will not re-establish even after splunk restart.
Connection gets re-established by editing outputs.conf and again closes with below logs after few hours.

05-23-2018 00:29:36.512 +0200 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
05-23-2018 00:29:36.512 +0200 INFO TcpOutputProc - Detected connection to 10.x.x.x:9997 closed
05-23-2018 00:29:36.512 +0200 INFO TcpOutputProc - Will close stream to current indexer 10.x.x.x:9997
05-23-2018 00:29:36.512 +0200 INFO TcpOutputProc - Closing stream for idx=10.x.x.x:9997
05-23-2018 00:29:36.767 +0200 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
05-23-2018 00:30:06.567 +0200 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed

0 Karma

renjith_nair
Legend

Hi @mravindra,

If its between forwarder and indexer, please check the ssl configuration. looks like your server is not trusting the certificate presented by client. It could be because of various reasons including host name validation, common name mismatch etc.

http://docs.splunk.com/Documentation/Splunk/7.1.1/Security/ConfigureSplunkforwardingtousesignedcerti...

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma

SasiB137
Engager

Right but the scenario here is "TCP connection closes after few hours and will not re-establish even after splunk restart." So not sure if it's a valid observation.

0 Karma

SasiB137
Engager

If we corrupt the forwarder certificate we should see the above logs getting emitted? Right ?

0 Karma
Get Updates on the Splunk Community!

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

🔐 Trust at Every Hop: How mTLS in Splunk Enterprise 10.0 Makes Security Simpler

From Idea to Implementation: Why Splunk Built mTLS into Splunk Enterprise 10.0  mTLS wasn’t just a checkbox ...