Getting Data In

Splunk forwarder crashes when scanned by Retina scanner

splunkIT
Splunk Employee
Splunk Employee

Splunk crashes frequently (not always) when scanned by Retina vulnerability scanning tool (http://www.eeye.com/products/retina/retina-network-scanner).

Something to do with the SSL handshake attempts that Retina is doing with the splunk service running on TCP/8080.
All splunk forwarders reported the SSL errors in splunkd.log, but not all crashed, but several did. This sounds like Retina can crash splunk, but not always.

Looking at splunkd.log just prior to crashing (the next log entries after these are related to splunk restarting):

08-29-2012 09:07:32.074 -0400 ERROR TcpInputFd - SSL Error for fd from HOST:172.28.41.199, IP:172.28.41.199, PORT:53924
08-29-2012 09:07:32.075 -0400 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:54
08-29-2012 09:07:32.075 -0400 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
08-29-2012 09:07:32.075 -0400 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
08-29-2012 09:07:32.075 -0400 ERROR TcpInputFd - SSL Error for fd from HOST:172.28.41.199, IP:172.28.41.199, PORT:53925
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:54
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL Error for fd from HOST:172.28.41.199, IP:172.28.41.199, PORT:53931
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:54
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL Error for fd from HOST:172.28.41.199, IP:172.28.41.199, PORT:53930

splunkIT
Splunk Employee
Splunk Employee

Splunk engineer has identified this to be a defect (SPL-55686), and they expect to have it fixed. Targeted on release 4.3.5 or later.

splunkIT
Splunk Employee
Splunk Employee

Looks like one or more of the firewall rules sent by Retina to the forwarder's splunkd (listening on port 8089) has triggered it to crash. This seems like a potential bug with the forwarder. The splunkd process should not be crashing because it has received an offending firewall rule front the scanner.

If you disable the forwarder's default management port (8089), the issue should go away. Universal forwarders do not necessary need to have the management port open to be functional.

Referred to this link to diable the management port:

http://splunk-base.splunk.com/answers/2371/can-i-disable-the-management-port-8089-in-a-light-forward...

splunkIT
Splunk Employee
Splunk Employee

This happens on universal forwarders for Linux and OS X.

yannK
Splunk Employee
Splunk Employee

On what OS and with which version was it happening ?

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...