Splunk crashes with ENGINE: Error in 'start' listener <bound method Server.start

[seyfried]

Hi there,

read about something like this a few times, but could not find any solution. Here's what happens: had splunk running fine. Then, I decided to re-install it, so I did a dpkg -P splunk, rm -Rf'd the remaining logs, re-installed via dpkg, started setting it up - which ends in a crashed splunkweb no matter what I do. I can do some clicks in the web interface before it dies with this error in web_service.log:

   2011-11-05 15:03:11,220 ERROR   [4eb541e76f145cf50] root:133 - ENGINE: Error in 'start' listener <bound method Server.start of <cherrypy._cpserver.Server object at 0x145ce50>>
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.6/site-packages/cherrypy/process/wspbus.py", line 147, in publish
    output.append(listener(*args, **kwargs))
  File "/opt/splunk/lib/python2.6/site-packages/cherrypy/_cpserver.py", line 93, in start
    ServerAdapter.start(self)
  File "/opt/splunk/lib/python2.6/site-packages/cherrypy/process/servers.py", line 60, in start
    self.wait()
  File "/opt/splunk/lib/python2.6/site-packages/cherrypy/process/servers.py", line 101, in wait
    wait_for_occupied_port(host, port)
  File "/opt/splunk/lib/python2.6/site-packages/cherrypy/process/servers.py", line 266, in wait_for_occupied_port
    raise IOError("Port %r not bound on %r" % (port, host))
IOError: Port 8000 not bound on '0.0.0.0'

What's a bit odd is the raised IOError - looks like the real error happens before that, because in that state, “splunk status” shows a missing spunkweb. Am I missing a log file here? I could produce the very same symptoms by either trying to install the *NIX app (without activating it, just downloading it from the splunkbase), or by trying to add /var/log as an input. Binding to the server IP instead of 0.0.0.0 did not help either.

I also tried to install the .tar.gz version instead of the dpkg, same results. Arch is AMD64, Apache running fine, as did splunk before... 😞

Any hints or suggestions?

Best,

Joe

[waynechenea]

I encountered a similar situation recently. Although the problem disappeared before I could debug it further.

Try running 'netstat -anb' and see what is binding port 8000. (Or whatever SplunkWeb is supposed to bind to).

Unfortunately, I was only running 'netstat -an' when I found that something had port 8000 stuck in TIME_WAIT. By the time I discovered 'netstat -anb' to see what process was stealing this port, it had already disappeared, and SplunkWeb managed to start.