How to resolve Splunk Web not starting after the hostname of our Splunk server changed?

[chadw72]

An admin recently changed the hostname of of our Splunk server and the user web interface stopped working. The splunkd service starts with no issues but the splunkweb service hangs. I changed the name back to its original but the web service will not bind. Here is a capture of what I'm seeing:

[servername]# /opt/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
...                                                        [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.

Splunk> The IT Search Engine.

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _internal _introspection _thefishbucket history main summary
        Done
        Checking filesystem compatibility...  Done
        Checking conf files for problems...
                Invalid key in stanza [ui] in /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/app.conf, line 15: attribution_link  (value:  app.attributions).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunk/splunk-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest'
File 'etc/system/default/transforms.conf' changed or missing.
        Problems were found, please review your files and move customizations to local
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
                                                           [  OK  ]

Waiting for web server at http://127.0.0.1:8001 to be available............  <---This will go on waiting forever


tail -50 /opt/splunk/var/log/splunk/web_service.log
2017-03-02 12:56:41,694 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Bus STOPPING
2017-03-02 12:56:43,760 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 8065)) shut down
2017-03-02 12:56:43,762 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Stopped thread '_TimeoutMonitor'.
2017-03-02 12:56:43,762 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Bus STOPPED
2017-03-02 12:56:43,763 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Bus EXITING
2017-03-02 12:56:43,763 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Bus EXITED
2017-03-02 12:56:43,763 ERROR   [58b85ca0557fe67f32ab90] root:810 - Unable to start splunkweb
2017-03-02 12:56:43,763 ERROR   [58b85ca0557fe67f32ab90] root:811 - Port 8065 not bound on '127.0.0.1'
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py", line 806, in 
    run(blocking=True)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py", line 687, in run
    cherrypy.engine.start()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 184, in start
    self.publish('start')
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 147, in publish
    output.append(listener(*args, **kwargs))
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cpserver.py", line 97, in start
    ServerAdapter.start(self)
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 61, in start
    self.wait()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 101, in wait
    wait_for_occupied_port(host, port)
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 266, in wait_for_occupied_port
    raise IOError("Port %r not bound on %r" % (port, host))
IOError: Port 8065 not bound on '127.0.0.1'

All the applicable firewall ports are open...it seems that 8000 and 8065 are not binding and I'm at a loss to figure out why. Any guidance would be appreciated.

Chad

[chadw72 · Accepted solution]

To all,
Issue has been resolved. I was port scanning the server during a reboot and even though Splunk was reporting 8089 and 8000 open, a port scanner never showed the ports active. I had to explicitly set the mgmtHostPort=x.x.x.x8089 and and the server.socket_host = x.x.x.x in the web.conf file to match the IP address in the /etc/hosts file. Once I did that splunkweb would activate and I can access the web console now.

[ndoshi]

Could it be that the ports for Splunk (8085 in your case, but 8000 is the default for Splunk Web) are still in use? If this is *nix, how about doing a ps -ef|grep splunk after you have done a splunk stop to see if there are any Splunk processes running and simply killing them. That may do it.

[adayton20]

Have you looked at this post?

https://answers.splunk.com/answers/7899/splunkweb-fails-to-start-timeout-when-binding-to-port.html

[adonio]

can you share /opt/splunk/etc/system/local/inputs.conf? under [default] is host = or
can you share /opt/splunk/etc/system/local/server.conf? under [general] what is serverName = to?
thanks!

[chadw72]

inputs.conf
[default]
host = servername.site1.local

server.conf
[general]
serverName = servername.site1.local

[chadw72]

BTW - servername is not the actual name but i have to redact that....suffice to say the entries match the hostname of the server

[adonio]

is this name reflect the change made by admin?

[chadw72]

All names within the config files match the host name once I reverted the hostname back

[adonio]

can you check splunkd.log /opt/splunk/var/log/splunk/splunkd.log for ERROR messages that might give a clue?