Getting Data In

Splunk forwarder crashes when scanned by Retina scanner

splunkIT
Splunk Employee
Splunk Employee

Splunk crashes frequently (not always) when scanned by Retina vulnerability scanning tool (http://www.eeye.com/products/retina/retina-network-scanner).

Something to do with the SSL handshake attempts that Retina is doing with the splunk service running on TCP/8080.
All splunk forwarders reported the SSL errors in splunkd.log, but not all crashed, but several did. This sounds like Retina can crash splunk, but not always.

Looking at splunkd.log just prior to crashing (the next log entries after these are related to splunk restarting):

08-29-2012 09:07:32.074 -0400 ERROR TcpInputFd - SSL Error for fd from HOST:172.28.41.199, IP:172.28.41.199, PORT:53924
08-29-2012 09:07:32.075 -0400 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:54
08-29-2012 09:07:32.075 -0400 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
08-29-2012 09:07:32.075 -0400 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
08-29-2012 09:07:32.075 -0400 ERROR TcpInputFd - SSL Error for fd from HOST:172.28.41.199, IP:172.28.41.199, PORT:53925
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:54
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL Error for fd from HOST:172.28.41.199, IP:172.28.41.199, PORT:53931
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:54
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
08-29-2012 09:07:32.077 -0400 ERROR TcpInputFd - SSL Error for fd from HOST:172.28.41.199, IP:172.28.41.199, PORT:53930

splunkIT
Splunk Employee
Splunk Employee

Splunk engineer has identified this to be a defect (SPL-55686), and they expect to have it fixed. Targeted on release 4.3.5 or later.

splunkIT
Splunk Employee
Splunk Employee

Looks like one or more of the firewall rules sent by Retina to the forwarder's splunkd (listening on port 8089) has triggered it to crash. This seems like a potential bug with the forwarder. The splunkd process should not be crashing because it has received an offending firewall rule front the scanner.

If you disable the forwarder's default management port (8089), the issue should go away. Universal forwarders do not necessary need to have the management port open to be functional.

Referred to this link to diable the management port:

http://splunk-base.splunk.com/answers/2371/can-i-disable-the-management-port-8089-in-a-light-forward...

splunkIT
Splunk Employee
Splunk Employee

This happens on universal forwarders for Linux and OS X.

yannK
Splunk Employee
Splunk Employee

On what OS and with which version was it happening ?

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...