Getting Data In

Why am I seeing TcpInputFd SSL Errors every 10 minutes on all of splunk instances?

Lowell
Super Champion

I've been seeing a bunch of error message sequences like this:

01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:0
01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
01-07-2011 09:45:40.106 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL Error for fd from HOST:splunk.my.domain, IP:192.168.1.99, PORT:3959

This is happening not only on my splunk indexer, but also on most (but not all) of my splunk forwarders too.

Observations:

  • The "HOST" and "IP" are always pointing to my central splunk indexes.
  • It seems like errno:0 could indicate that there is NO error (since low-level OS calls often use a return code 0 to mean "success", and any other value indicates a specific error code.)
  • Sometime I see SSL_ERROR_SYSCALL ret errno:104 instead of errono:0 but that occurs much less frequently (and the subsequent errors show above are the same)

Any ideas? Should I be concerned about these?

1 Solution

Lowell
Super Champion

After some network packet tracking with wireshark and a bunch of head scratching; I've finally found the answer:

The messages are a result of the "Splunk Monitoring" app that I installed some time ago and had forgotten about. The app is setup to poll a given list of splunk servers (which was out of date on my system, hence not ALL of my splunk instances were reporting this.) The app uses a scripted input that runs on a regular interval (10 minutes on my system.) The basic ping-like approach used by the app actually uses nmap to look for a response on port 8089 (which is the the splunkd HTTPS listener). So because there is no actual request being made, nothing gets logged in the splunkd_access log, and because there is no attempt to handle the SSL stuff, you end up with errors coming from splunkd about an invalid connection. Which is accurate.

So the bottom line is that these messages are not harmful, but there may be a better way to monitor for up/down activity that using nmap. Perhaps there is some internal "ping" like URL that could be used with wget or curl or something like that.


Since this took a while track down, I figure it was worth sharing. And of course any other kind of port scanning device or simple TCP monitoring service would most likely also trigger this kind of messages too. Perhaps this will save somebody some time.

View solution in original post

bohrasaurabh
Communicator

In our environment we saw a lot of ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:104 on our Deployment server for the forwarders sitting behind DMZ. We requested our security team to open the firewall and the error messages disappeared.

0 Karma

Lowell
Super Champion

After some network packet tracking with wireshark and a bunch of head scratching; I've finally found the answer:

The messages are a result of the "Splunk Monitoring" app that I installed some time ago and had forgotten about. The app is setup to poll a given list of splunk servers (which was out of date on my system, hence not ALL of my splunk instances were reporting this.) The app uses a scripted input that runs on a regular interval (10 minutes on my system.) The basic ping-like approach used by the app actually uses nmap to look for a response on port 8089 (which is the the splunkd HTTPS listener). So because there is no actual request being made, nothing gets logged in the splunkd_access log, and because there is no attempt to handle the SSL stuff, you end up with errors coming from splunkd about an invalid connection. Which is accurate.

So the bottom line is that these messages are not harmful, but there may be a better way to monitor for up/down activity that using nmap. Perhaps there is some internal "ping" like URL that could be used with wget or curl or something like that.


Since this took a while track down, I figure it was worth sharing. And of course any other kind of port scanning device or simple TCP monitoring service would most likely also trigger this kind of messages too. Perhaps this will save somebody some time.

swhiting
Engager

I know this thread is ancient, but wanted to ask Jason if he ever figured out if the ssl errors related to his deployment server issues?

Jason
Motivator

I am also seeing these errors, identical except for errno is errno: 104 - and we're seeing horrible performance in Deployment Server. We're still investigating.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...