Why am I getting "Error in HTTP server: shutting down" when starting splunkweb?

[letsrumbl]

Hello all !

Environment:
- OS - Ubuntu 12.04.6
- Splunk - splunk-6.1.3-220630-linux-2.6-amd64.deb
- Installed via dpkg
- Independent installation

I'm having an issue with starting splunkweb on the first go.

Here is the error from web_services.log

2014-10-03 19:17:09,447 INFO    [542ef6340610d2450] root:138 - ENGINE: Bus STARTING
2014-10-03 19:17:09,475 INFO    [542ef6340610d2450] root:138 - ENGINE: Started monitor thread '_TimeoutMonitor'.
2014-10-03 19:17:09,492 ERROR   [542ef6340610d2450] root:138 - ENGINE: Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 73, in _start_http_thread
    self.httpserver.start()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/wsgiserver/__init__.py", line 1725, in start
    raise socket.error, msg
error: [Errno 13] Permission denied

2014-10-03 19:17:09,492 INFO    [542ef6340610d2450] root:138 - ENGINE: Bus STOPPING
2014-10-03 19:17:09,492 ERROR   [542ef6340610d2450] root:138 - ENGINE: Error in 'stop' listener <bound method Server.stop of <cherrypy._cpserver.Server object at 0x10d2350>>
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 147, in publish
    output.append(listener(*args, **kwargs))
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 107, in stop
    self.httpserver.stop()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/wsgiserver/__init__.py", line 1909, in stop
    self.requests.stop(self.shutdown_timeout)
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/wsgiserver/__init__.py", line 1458, in stop
    self._monitor_thread.join()
AttributeError: 'NoneType' object has no attribute 'join'

2014-10-03 19:17:09,494 INFO    [542ef6340610d2450] root:138 - ENGINE: Stopped thread '_TimeoutMonitor'.

To replicate:

Any time I restart splunk (/etc/init.d/splunk restart OR $SPLUNK_HOME/bin/splunk restart) the splunkd restarts fine but web_services.log throws the error above.

If I follow the restart with a splunk start splunkweb the web server comes up with no errors. Main problem with this is that on system start, web server does not fire off and I have to manually log in to start it.

Tried:
- Reinstalling package
- Changing permissions
- Changed to root (to test)
- Also verified port is not in use before starting (shouldn't matter on restart)

Anyone encounter this issue ?

[grijhwani]

Looks to me like you previously started Splunk as root (started on boot?), and now you are trying to start it as some unpriveleged user (perhaps the splunk user itself) as a manual start.

[letsrumbl]

When I added it to the startup scripts I tried both combinations (as root user and as splunk).

Regardless of what I did both scenarios yielded the same outcome.

I believe that if it was a permissions issue, and I were to run splunk start splunkweb as root, it would start as expected. No matter which user I use, the first time I run splunk start splunkweb it renders in the error above. As I explained above, if this is followed by the same command immediately after, the service starts correctly.