https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15975#M449 <P>As a short-term remedy... I set up CRON to restart service once a day...</P> Wed, 23 Jun 2010 08:14:17 GMT balbano 2010-06-23T08:14:17Z https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15972#M446 <P>Hey guys, </P> <P>Doesn't seem like many people have had problems with the Splunk Web Service hanging on them, but this is somewhat similar to: </P> <P><A href="http://answers.splunk.com/questions/1468/splunkweb-crashes-although-splunkd-is-still-running-is-this-a-known-problem" rel="nofollow">http://answers.splunk.com/questions/1468/splunkweb-crashes-although-splunkd-is-still-running-is-this-a-known-problem</A></P> <P>Basically, I have been testing my Foundstone Vulnerability Scanner on my splunk indexers and search head and it looks like at least one of servers has its Splunk Web Service die. Port is reachable via telnet, and splunk status states that the service is up, but the site itself is unreachable. </P> <P>I simulated the situation and tailed the logs and see these types of errors:</P> <H2>splunkd.log</H2> <P>06-22-2010 20:42:42.816 ERROR NetUtils - SSL_ERROR_SSL in SSL_write. nbytes=-1, Error = error:140D00CF:SSL routines:SSL_write:protocol is shutdown</P> <P>06-22-2010 20:42:43.210 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request<BR /> 06-22-2010 20:42:43.210 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0<BR /> 06-22-2010 20:42:43.210 ERROR TcpInputFd - SSL Error for fd from HOST:$foundstone_scanner$, IP:$foundstone_IP$, PORT:5927<BR /> 06-22-2010 20:42:43.365 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request<BR /> 06-22-2010 20:42:43.365 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0<BR /> 06-22-2010 20:42:43.365 ERROR TcpInputFd - SSL Error for fd from HOST:$foundstone_scanner$, IP:$foundstone_IP$, PORT:5939<BR /> 06-22-2010 20:42:44.116 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request<BR /> 06-22-2010 20:42:44.116 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0 </P> <P>06-22-2010 20:45:11.376 INFO TailingProcessor - File descriptor cache is full (64), trimming...<BR /> 06-22-2010 20:45:17.397 INFO TailingProcessor - File descriptor cache is full (64), trimming...<BR /> 06-22-2010 20:45:19.516 INFO TailingProcessor - File descriptor cache is full (64), trimming...<BR /> 06-22-2010 20:46:22.830 INFO TailingProcessor - File descriptor cache is full (64), trimming... </P> <H2>web_service.log</H2> <P>This is the error message that appears right before it hangs:</P> <P>2010-06-18 19:05:05,842 ERROR [4c1798223163010d0] root:120 - ENGINE: Error in HTTP server: shutting down<BR /> Traceback (most recent call last):<BR /> File "/opt/splunk/lib/python2.6/site-packages/cherrypy/process/servers.py", line 73, in _start_http_thread<BR /> File "/opt/splunk/lib/python2.6/site-packages/cherrypy/wsgiserver/<STRONG>init</STRONG>.py", line 1662, in start<BR /> self.tick()<BR /> File "/opt/splunk/lib/python2.6/site-packages/cherrypy/wsgiserver/<STRONG>init</STRONG>.py", line 717, in tick<BR /> s, addr = self.socket.accept()<BR /> File "/opt/splunk/lib/python2.6/ssl.py", line 317, in accept<BR /> newsock, addr = socket.accept(self)<BR /> File "/opt/splunk/lib/python2.6/socket.py", line 195, in accept<BR /> error: [Errno 24] Too many open files </P> <P>Anyone ever have the same issue?</P> <P>I also ran splunk diag so that I can open support ticket but just wanted to see if anyone else has ever had this problem. </P> <P>Let me know. </P> <P>Thanks Guys!</P> <P>Brian</P> Wed, 23 Jun 2010 06:22:13 GMT https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15972#M446 balbano 2010-06-23T06:22:13Z https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15973#M447 <P>Can you check 2 things for me:</P> <P>1) does a ./splunk restart splunkweb fix the hang? (or does it time out) 2) Can you run searches in CLI mode?</P> <P>I have definitely seen this issue come up, but am unsure if it is exactly the same issue or not, however, if the two above happen then we already have a fix for it. Contacting support would probably be the best way for you to get this issue troubleshooted further, and receive the new patch.</P> <P>Let me know on the above two conditions...<BR /> Cheers,<BR /> .gz</P> Wed, 23 Jun 2010 07:22:24 GMT https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15973#M447 Genti 2010-06-23T07:22:24Z https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15974#M448 <OL> <LI><P>restarting web service fixes the problem...</P></LI> <LI><P>Never tried searches in CLI mode since the service restart fixes the issue.... I can try and get back to you on that...</P></LI> </OL> <P>Thanks</P> <P>Brian</P> Wed, 23 Jun 2010 08:13:45 GMT https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15974#M448 balbano 2010-06-23T08:13:45Z https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15975#M449 <P>As a short-term remedy... I set up CRON to restart service once a day...</P> Wed, 23 Jun 2010 08:14:17 GMT https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15975#M449 balbano 2010-06-23T08:14:17Z https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15976#M450 <P>Filed support ticket with Splunk Support and Patch was applied to remedy the issue. If anyone has a similar issue you should file support case and have the symptoms diagnosed by the Splunk Engineers. </P> <P>Thanks.</P> <P>Brian</P> Thu, 24 Jun 2010 21:08:58 GMT https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15976#M450 balbano 2010-06-24T21:08:58Z https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15977#M451 <P>Mine only appears during vulnerability scanning.</P> Fri, 03 May 2013 17:29:29 GMT https://community.splunk.com/t5/Security/SplunkWeb-Service-occasionally-hangs/m-p/15977#M451 Strype 2013-05-03T17:29:29Z