Unable to bind splunk to IP

[atewari]

We would like to access Splunk Web from other hosts. We did a full splunk 5.0.1 (build: 143156) install on a Windows Server 2008 64-bit. Everything worked fine with default. We could access the Splunk portal locally from the Windows box ONLY using http://localhost:8000. However, we could not use the machine's IP (10.10.10.10) as http://10.10.10.10:8000. As per Splunk's documentation on http://docs.splunk.com/Documentation/Splunk/latest/Admin/BindSplunktoanIP, we made the following changes and restarted SplunnkD and SplunkWeb with no luck. We primarily want to access Splunk Web from other servers. We are successfully able to ping the Splunk server from these other servers.

web.conf

# location of splunkd; don't include http[s]:// in this anymore.
# mgmtHostPort = 127.0.0.1:8089
# commented the above line to enforce IP
mgmtHostPort = 10.10.10.10:8089

# enforce the IP
server.socket_host = 10.10.10.10

splunk-launch.conf

# If unset, Splunk will use the parent of the directory this configuration
# file was found in
#
# SPLUNK_HOME=c:\build-amd64-5.0.1-20121112-1621\splunk

#  Enforcing IP
SPLUNK_BINDIP=10.10.10.10

The SplunkWeb server goes down after attempting to run, when we restart with web_service.log

2012-11-27 15:51:00,614 INFO    [50b535aea622cf390] root:535 - CONFIG: ui_inactivity_timeout (int): 60
2012-11-27 15:51:00,614 INFO    [50b535aea622cf390] root:535 - CONFIG: updateCheckerBaseURL (str): https://quickdraw.splunk.com/js/
2012-11-27 15:51:00,614 INFO    [50b535aea622cf390] root:535 - CONFIG: use_future_expires (bool): True
2012-11-27 15:51:00,614 INFO    [50b535aea622cf390] root:535 - CONFIG: userRegistrationURL (str): https://www.splunk.com/index.php/pre_reg?destination=prod_reg
2012-11-27 15:51:00,614 INFO    [50b535aea622cf390] root:535 - CONFIG: version_label (str): UNKNOWN_VERSION
2012-11-27 15:51:00,614 INFO    [50b535aea622cf390] root:535 - CONFIG: version_number (str): 4.0
2012-11-27 15:51:00,614 INFO    [50b535aea622cf390] root:134 - ENGINE: Bus STARTING
2012-11-27 15:51:00,630 INFO    [50b535aea622cf390] root:134 - ENGINE: Set handler for console events.
2012-11-27 15:51:00,630 INFO    [50b535aea622cf390] root:134 - ENGINE: Started monitor thread '_TimeoutMonitor'.
2012-11-27 15:51:57,023 ERROR   [50b535aea622cf390] root:134 - ENGINE: Error in 'start' listener <bound method Server.start of <cherrypy._cpserver.Server object at 0x00000000022A5940>>
Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\process\wspbus.py", line 147, in publish
    output.append(listener(*args, **kwargs))
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\_cpserver.py", line 94, in start
    ServerAdapter.start(self)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\process\servers.py", line 60, in start
    self.wait()
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\process\servers.py", line 101, in wait
    wait_for_occupied_port(host, port)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\process\servers.py", line 266, in wait_for_occupied_port
    raise IOError("Port %r not bound on %r" % (port, host))
IOError: Port 8000 not bound on '10.10.10.10'

2012-11-27 15:51:57,023 ERROR   [50b535aea622cf390] root:134 - ENGINE: Shutting down due to error in start listener:
Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\process\wspbus.py", line 184, in start
    self.publish('start')
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\process\wspbus.py", line 147, in publish
    output.append(listener(*args, **kwargs))
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\_cpserver.py", line 94, in start
    ServerAdapter.start(self)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\process\servers.py", line 60, in start
    self.wait()
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\process\servers.py", line 101, in wait
    wait_for_occupied_port(host, port)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\cherrypy\process\servers.py", line 266, in wait_for_occupied_port
    raise IOError("Port %r not bound on %r" % (port, host))
IOError: Port 8000 not bound on '10.10.10.10'

2012-11-27 15:51:57,023 INFO    [50b535aea622cf390] root:134 - ENGINE: Bus STOPPING
2012-11-27 15:51:57,023 INFO    [50b535aea622cf390] root:134 - ENGINE: HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('10.10.10.10', 8000)) already shut down
2012-11-27 15:51:57,023 INFO    [50b535aea622cf390] root:134 - ENGINE: Stopped thread '_TimeoutMonitor'.
2012-11-27 15:51:57,023 INFO    [50b535aea622cf390] root:134 - ENGINE: Removed handler for console events.
2012-11-27 15:51:57,023 INFO    [50b535aea622cf390] root:134 - ENGINE: Bus STOPPED
2012-11-27 15:51:57,023 INFO    [50b535aea622cf390] root:134 - ENGINE: Bus EXITING
2012-11-27 15:51:57,023 INFO    [50b535aea622cf390] root:134 - ENGINE: Bus EXITED

[tasdienes]

Hi atewari, I am having the same problem. I'm trying to run Splunk on a server that is also running DirectAccess. Can you explain a bit more about the problem/solution?
Thanks!

[atewari]

It appears the IPsec on Windows Server 2008 was disallowing. Once that was enabled, it worked beautifully. Thanks Ayn for your quick responses. Awarding point shortly.

[atewari]

Thanks Ayn. I ran the netstat -nao command and noticed that there were entries. We added the forwarding port 8001.

Proto  Local Address          Foreign Address        State           PID
TCP    0.0.0.0:8000           0.0.0.0:0              LISTENING       1856
TCP    0.0.0.0:8001           0.0.0.0:0              LISTENING       3920
TCP    0.0.0.0:8089           0.0.0.0:0              LISTENING       3920

But there are no entries for these ports like port 80, which has [::]

Proto  Local Address          Foreign Address        State           PID

TCP    0.0.0.0:80             0.0.0.0:0              LISTENING       4

TCP    [::]:80                [::]:0                 LISTENING       4

Why is that? Could this be the cause of unable to bind IP on port 8000?

[atewari · Accepted solution]

Ayn,
It appears that Splunk, by default, does not listen on all interfaces. I ran a simple test with IIS. I started IIS and then did telnet localhost 80 and then telnet 10.10.10.10 80. In both cases, it connected.

But when I did this for Splunk:

• telnet localhost 8000 - this worked
• telnet 10.10.10.10 8000 - this did not connect.

Clearly, on windows 64-bit, Splunk 5.0.1 some configuration is different. Any ideas are much appreciated.

Thanks!

[Ayn]

I beg to differ - this is from my Splunk 5.0 install:

# lsof -n -i -P | grep splunk
splunkd    6434     root    4u  IPv4 594359      0t0  TCP *:8089 (LISTEN)

The docs state the same - default behaviour is to listen to all interfaces. If you want to verify this in your Windows install you can do a netstat -nao and check which ports the PID belonging to Splunk listens to.

[atewari]

Thanks Ayn and Drainy.

We thought it was a firewall issue too, so we disabled it all together. We reverted to default changes, disabled the firewall, and then restarted SplunkD and SplunkWeb. We can connect from the local box using http://localhost:8000, but cannot connect using the IP on the local box http://10.10.10.10:8000.

So we did a simple test - we enabled IIS 7.0 to ensure we can reach IIS on port 80 from other servers. It worked fine - http://10.10.10.10.

Any ideas why SplunkWeb won't respond locally with an IP (http://10.10.10.10:8000). I'm assuming, if the local server works, we should be able to connect remotely with the firewall disabled.

[Ayn]

By default Splunk will listen to all interfaces, so for troubleshooting purposes I think the best option is to revert the changes you made and go back to the basic problem which is why you're not able to connect to the splunkweb port from any other machines than localhost. Sounds like a firewall problem to me.

[Drainy]

Yup, sounds like a firewall issue if you can access it via localhost. Do what Ayn says, he speaks sense.