Security

How to resolve Splunk Web not starting after the hostname of our Splunk server changed?

Explorer

An admin recently changed the hostname of of our Splunk server and the user web interface stopped working. The splunkd service starts with no issues but the splunkweb service hangs. I changed the name back to its original but the web service will not bind. Here is a capture of what I'm seeing:

[servername]# /opt/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
...                                                        [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.

Splunk> The IT Search Engine.

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _internal _introspection _thefishbucket history main summary
        Done
        Checking filesystem compatibility...  Done
        Checking conf files for problems...
                Invalid key in stanza [ui] in /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/app.conf, line 15: attribution_link  (value:  app.attributions).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunk/splunk-6.3.1-f3e41e4b37b2-linux-2.6-x86_64-manifest'
File 'etc/system/default/transforms.conf' changed or missing.
        Problems were found, please review your files and move customizations to local
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
                                                           [  OK  ]

Waiting for web server at http://127.0.0.1:8001 to be available............  <---This will go on waiting forever


tail -50 /opt/splunk/var/log/splunk/web_service.log
2017-03-02 12:56:41,694 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Bus STOPPING
2017-03-02 12:56:43,760 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 8065)) shut down
2017-03-02 12:56:43,762 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Stopped thread '_TimeoutMonitor'.
2017-03-02 12:56:43,762 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Bus STOPPED
2017-03-02 12:56:43,763 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Bus EXITING
2017-03-02 12:56:43,763 INFO    [58b85ca0557fe67f32ab90] root:129 - ENGINE: Bus EXITED
2017-03-02 12:56:43,763 ERROR   [58b85ca0557fe67f32ab90] root:810 - Unable to start splunkweb
2017-03-02 12:56:43,763 ERROR   [58b85ca0557fe67f32ab90] root:811 - Port 8065 not bound on '127.0.0.1'
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py", line 806, in 
    run(blocking=True)
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py", line 687, in run
    cherrypy.engine.start()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 184, in start
    self.publish('start')
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 147, in publish
    output.append(listener(*args, **kwargs))
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cpserver.py", line 97, in start
    ServerAdapter.start(self)
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 61, in start
    self.wait()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 101, in wait
    wait_for_occupied_port(host, port)
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 266, in wait_for_occupied_port
    raise IOError("Port %r not bound on %r" % (port, host))
IOError: Port 8065 not bound on '127.0.0.1'

All the applicable firewall ports are open...it seems that 8000 and 8065 are not binding and I'm at a loss to figure out why. Any guidance would be appreciated.

Chad

0 Karma
1 Solution

Explorer

To all,
Issue has been resolved. I was port scanning the server during a reboot and even though Splunk was reporting 8089 and 8000 open, a port scanner never showed the ports active. I had to explicitly set the mgmtHostPort=x.x.x.x8089 and and the server.socket_host = x.x.x.x in the web.conf file to match the IP address in the /etc/hosts file. Once I did that splunkweb would activate and I can access the web console now.

View solution in original post

0 Karma

Explorer

To all,
Issue has been resolved. I was port scanning the server during a reboot and even though Splunk was reporting 8089 and 8000 open, a port scanner never showed the ports active. I had to explicitly set the mgmtHostPort=x.x.x.x8089 and and the server.socket_host = x.x.x.x in the web.conf file to match the IP address in the /etc/hosts file. Once I did that splunkweb would activate and I can access the web console now.

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

Could it be that the ports for Splunk (8085 in your case, but 8000 is the default for Splunk Web) are still in use? If this is *nix, how about doing a ps -ef|grep splunk after you have done a splunk stop to see if there are any Splunk processes running and simply killing them. That may do it.

0 Karma

Contributor
0 Karma

SplunkTrust
SplunkTrust

can you share /opt/splunk/etc/system/local/inputs.conf? under [default] is host = or
can you share /opt/splunk/etc/system/local/server.conf? under [general] what is serverName = to?
thanks!

0 Karma

Explorer

inputs.conf
[default]
host = servername.site1.local

server.conf
[general]
serverName = servername.site1.local

0 Karma

Explorer

BTW - servername is not the actual name but i have to redact that....suffice to say the entries match the hostname of the server

0 Karma

SplunkTrust
SplunkTrust

is this name reflect the change made by admin?

0 Karma

Explorer

All names within the config files match the host name once I reverted the hostname back

0 Karma

SplunkTrust
SplunkTrust

can you check splunkd.log /opt/splunk/var/log/splunk/splunkd.log for ERROR messages that might give a clue?

0 Karma