Splunk 4.1.4 on Win2008 R2, splunkweb is repeatedly terminating with "Bad file descriptor" error shown in web_serivce.log, requiring a restart of the service. A uninstall/reinstall has not resolved this issue.
Any known issues, solutions?
2010-11-09 16:25:35,484 ERROR [4cd975ff585418320] module:62 - [Errno 9] Bad file descriptor Traceback (most recent call last): File "F:\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\controllers\module.py", line 60, in renderModule pageContent = instance.generateResults(**args) File "F:\Splunk\share\splunk\search_mrsparkle\modules\results\SimpleResultsTable.py", line 137, in generateResults adjustedMin, adjustedMax = util.getPercentiles(ordered, .05, .95) File "F:\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 454, in getPercentiles return (v(lowerPercentile, orderedList), v(upperPercentile, orderedList)) File "F:\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 446, in v n, k, d = f(percentile, len(oList)) File "F:\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 442, in f print n,k,d IOError: [Errno 9] Bad file descriptor 2010-11-09 16:36:32,776 ERROR [4cd96d628f2342080] root:120 - ENGINE: Error in HTTP server: shutting down Traceback (most recent call last): File "F:\Splunk\Python-2.6\Lib\site-packages\cherrypy\process\servers.py", line 73, in _start_http_thread self.httpserver.start() File "F:\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1665, in start self.tick() File "F:\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1730, in tick prevent_socket_inheritance(s) File "F:\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1474, in prevent_socket_inheritance raise WinError() WindowsError: [Error 6] The handle is invalid. 2010-11-09 16:36:32,792 INFO [4cd96d628f2342080] root:120 - ENGINE: Bus STOPPING 2010-11-09 16:36:51,809 INFO [4cd96d628f2342080] root:120 - ENGINE: HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8000)) shut down 2010-11-09 16:36:51,825 INFO [4cd96d628f2342080] root:120 - ENGINE: Stopped thread 'Monitor'. 2010-11-09 16:36:51,855 INFO [4cd96d628f2342080] root:120 - ENGINE: Stopped thread 'Monitor'. 2010-11-09 16:36:51,917 INFO [4cd96d628f2342080] root:120 - ENGINE: Stopped thread 'Monitor'. 2010-11-09 16:36:51,980 INFO [4cd96d628f2342080] root:120 - ENGINE: Stopped thread '_TimeoutMonitor'. 2010-11-09 16:36:51,980 ERROR [4cd96d628f2342080] root:120 - ENGINE: Removed handler for console events. 2010-11-09 16:36:51,980 WARNING [4cd96d628f2342080] custompidfile:36 - Removing pidfile at F:\Splunk\var\run\splunk\splunkweb.pid (PID: 8080)" 2010-11-09 16:36:51,980 INFO [4cd96d628f2342080] root:120 - ENGINE: Bus STOPPED 2010-11-09 16:36:51,980 INFO [4cd96d628f2342080] root:120 - ENGINE: Bus EXITING 2010-11-09 16:36:51,980 INFO [4cd96d628f2342080] root:120 - ENGINE: Bus EXITED
See:
http://www.cherrypy.org/ticket/1016
Please reply back here if applying the patch to the specified file proves successful.
Are there any updates for this issue? I'm currently seeing this on a very busy Enterprise Security Search head. It seems to happen about 20 times in a month period. Would this get resolved by increasing the ulimit settings on the linux server, or is there some limits.conf setting that should be looked at on the search head? This is Splunk Enterprise 6.2, so it is still happening on newer versions. One other caveat, it resides in amazon cloud.,
Same here - any news?
I have exactly the same thing. Running Splunk 4.1.6 on Windows Server 2008 R2 64-bit, sometimes when I change my search rapidly (edit query it in the middle of a search and re-start the search, repeat a couple of times), Splunk Web crashes. Very annoying. This is the error in web_service.log:
2010-12-03 12:37:27,052 ERROR [4cf8bcbcda1ee6208] root:141 - ENGINE: Error in HTTP server: shutting down
Traceback (most recent call last):
File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\cherrypy\process\servers.py", line 73, in _start_http_thread
self.httpserver.start()
File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1665, in start
self.tick()
File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1730, in tick
prevent_socket_inheritance(s)
File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1474, in prevent_socket_inheritance
raise WinError()
WindowsError: [Error 6] The handle is invalid.
I have applied the patch and will note in a couple of days whether the crash recurs.
in case it helps -- i've seen that 'bad file descriptor' stack trace too once in a while on 4.1.5, but I've never seen it trigger a shutdown.
See:
http://www.cherrypy.org/ticket/1016
Please reply back here if applying the patch to the specified file proves successful.
This particular link is now inactive. Any information on a current link?
Thanks for reporting back. As noted in the bug, what a weird fix!
Based on previous history of multiple crashes per day, and a full day since patching without issue, it looks like this was the solution.
Patch applied. If I don't get another crash in a day, I will assume issue is fixed. Will report my findings.