I've been seeing a bunch of error message sequences like this:
01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:0
01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
01-07-2011 09:45:40.106 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL Error for fd from HOST:splunk.my.domain, IP:192.168.1.99, PORT:3959
This is happening not only on my splunk indexer, but also on most (but not all) of my splunk forwarders too.
Observations:
errno:0
could indicate that there is NO error (since low-level OS calls often use a return code 0 to mean "success", and any other value indicates a specific error code.)SSL_ERROR_SYSCALL ret errno:104
instead of errono:0
but that occurs much less frequently (and the subsequent errors show above are the same)Any ideas? Should I be concerned about these?
After some network packet tracking with wireshark and a bunch of head scratching; I've finally found the answer:
The messages are a result of the "Splunk Monitoring" app that I installed some time ago and had forgotten about. The app is setup to poll a given list of splunk servers (which was out of date on my system, hence not ALL of my splunk instances were reporting this.) The app uses a scripted input that runs on a regular interval (10 minutes on my system.) The basic ping-like approach used by the app actually uses nmap
to look for a response on port 8089 (which is the the splunkd
HTTPS listener). So because there is no actual request being made, nothing gets logged in the splunkd_access log, and because there is no attempt to handle the SSL stuff, you end up with errors coming from splunkd about an invalid connection. Which is accurate.
So the bottom line is that these messages are not harmful, but there may be a better way to monitor for up/down activity that using nmap
. Perhaps there is some internal "ping" like URL that could be used with wget
or curl
or something like that.
Since this took a while track down, I figure it was worth sharing. And of course any other kind of port scanning device or simple TCP monitoring service would most likely also trigger this kind of messages too. Perhaps this will save somebody some time.
In our environment we saw a lot of ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:104 on our Deployment server for the forwarders sitting behind DMZ. We requested our security team to open the firewall and the error messages disappeared.
After some network packet tracking with wireshark and a bunch of head scratching; I've finally found the answer:
The messages are a result of the "Splunk Monitoring" app that I installed some time ago and had forgotten about. The app is setup to poll a given list of splunk servers (which was out of date on my system, hence not ALL of my splunk instances were reporting this.) The app uses a scripted input that runs on a regular interval (10 minutes on my system.) The basic ping-like approach used by the app actually uses nmap
to look for a response on port 8089 (which is the the splunkd
HTTPS listener). So because there is no actual request being made, nothing gets logged in the splunkd_access log, and because there is no attempt to handle the SSL stuff, you end up with errors coming from splunkd about an invalid connection. Which is accurate.
So the bottom line is that these messages are not harmful, but there may be a better way to monitor for up/down activity that using nmap
. Perhaps there is some internal "ping" like URL that could be used with wget
or curl
or something like that.
Since this took a while track down, I figure it was worth sharing. And of course any other kind of port scanning device or simple TCP monitoring service would most likely also trigger this kind of messages too. Perhaps this will save somebody some time.
I know this thread is ancient, but wanted to ask Jason if he ever figured out if the ssl errors related to his deployment server issues?
I am also seeing these errors, identical except for errno
is errno: 104
- and we're seeing horrible performance in Deployment Server. We're still investigating.