I see some of these time outs in the /var/log/splunk/splunk.log
Is this something I should be concerned about? Does the forwarder try a resend? Is this a potential data loss? or if there's a retry, does it handle the resend gracefully?
01-13-2012 01:52:56.098 +0000 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 01-13-2012 01:54:15.760 +0000 WARN TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out 01-13-2012 01:54:15.762 +0000 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 01-13-2012 01:54:45.592 +0000 WARN TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out 01-13-2012 01:55:15.423 +0000 WARN TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out 01-13-2012 01:55:15.424 +0000 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 01-13-2012 01:55:34.333 +0000 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997
If the issue is persistent, I suspect your forwarder is configured to setup an unencrypted connection to the indexer but the indexer only accepts encrypted connections - or vice versa.
Thank you for your response, Ayn. I have both forwarder and receiver to use ssl... and I have done it many times... not sure what is different this time.
I have case opened, will see:
Description: WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9992 timed out
-bash-3.2# cat outputs.conf
defaultGroup = splunkssl-LB
server = splunk06:9992
compressed = true
sslCertPath = $SPLUNKHOME/etc/certs/forwarder.pem
sslCommonNameToCheck = indexer
sslPassword = xxxxxxxxxxxxxxxx
sslRootCAPath = $SPLUNKHOME/etc/certs/cacert.pem
sslVerifyServerCert = true
host = splunk06
password = xxxxxxxxxxxxxxxxxxx
requireClientCert = true
rootCA = $SPLUNKHOME/etc/certs/cacert.pem
serverCert = $SPLUNKHOME/etc/certs/indexer.pem
compressed = true
I do not understand how this solved the issue. I have the issue sometimes (as seems to be the case in your question), that is, not always. Simply a misisng certificate would mean the problem should always happen, right?
I just setup my new implementation and I am getting this error message intermittently for one of the three indexers, which is randomly "picked" to report this error. Forwarders and Indexers are otherwise communicating properly. What else might I look for as possible cause for this situation?