topic Re: Splunk instances will not connect to HTTP port in Security

Splunk instances will not connect to HTTP port

jonesnadiam — Thu, 06 Apr 2017 17:43:21 GMT

After installing SSL certificates and changing the default Splunk web port to 443, I receive the following error:

Checking http port [443]: already bound
ERROR: The http port [443] is already bound. Splunk needs to use this port.

After killing the processes associated with this port and rebooting, I am still unable to start Splunk, receiving the same error. I've also tried the following with no luck:

Confirmed the loopback address in /etc/hosts and ifconfig lo
Confirmed there was no BIND_IP defined in /opt/splunk/etc/splunk-launch.conf
Tried removing/re-installing Splunk

Any other suggestions?

Note - I have 5 instances of Splunk (1DS, 2SH, 2HF). The only instance that was able to successfully connect to the port was the DS.

Re: Splunk instances will not connect to HTTP port

suarezry — Thu, 06 Apr 2017 17:52:34 GMT

Is the host windows or linux? From the host try 'telnet localhost 443' to see if that port is in use.

Re: Splunk instances will not connect to HTTP port

jonesnadiam — Thu, 06 Apr 2017 18:13:46 GMT

This is a Linux server. I receiving the following:

Trying ::1...
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Re: Splunk instances will not connect to HTTP port

suarezry — Thu, 06 Apr 2017 19:25:25 GMT

Ok, so port 443 is free. What user are you trying to run splunk as? Port 443 is a privileged port.

Re: Splunk instances will not connect to HTTP port

jonesnadiam — Thu, 06 Apr 2017 19:36:14 GMT

Yep - I am running as root 😞

Re: Splunk instances will not connect to HTTP port

suarezry — Thu, 06 Apr 2017 19:41:05 GMT

post your $SPLUNK_HOME/etc/system/local/web.conf and server.conf

Re: Splunk instances will not connect to HTTP port

jonesnadiam — Thu, 06 Apr 2017 19:50:50 GMT

web.conf:

[settings]
httpport = 443
enableSplunkWebSSL = true
privKeyPath =
serverCert =

server.conf:
[general]
serverName = .domain.com

[sslConfig]
sslPassword =

Re: Splunk instances will not connect to HTTP port

suarezry — Thu, 06 Apr 2017 19:56:50 GMT

Try changing the port to 8443 just to confirm it starts up fine and not a config issue.

Re: Splunk instances will not connect to HTTP port

jonesnadiam — Thu, 06 Apr 2017 20:05:45 GMT

Yep, that works fine:

Checking prerequisites...
Checking http port [8443]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
[...]
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

Waiting for web server at https://127.0.0.1:8443 to be available..........

Re: Splunk instances will not connect to HTTP port

suarezry — Thu, 06 Apr 2017 20:17:36 GMT

What is the output of these 2 commands:

netstat -na|grep 443
lsof -i|grep 443

Re: Splunk instances will not connect to HTTP port

jonesnadiam — Fri, 07 Apr 2017 01:50:17 GMT

When running the netstat command, I get the following:
tcp 0 1 :55742 :443 SYN_SENT

When running the lsof command, I get the following:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
connector 3899 root 6u IPv4 92461 0t0 TCP servername.domain.com:58607->name.domain.com:https (SYN_SENT)

Re: Splunk instances will not connect to HTTP port

suarezry — Fri, 07 Apr 2017 13:49:42 GMT

Wow ok...So you confirmed nothing is bound to port 443, your loopback and splunk-launch.conf is good, and you are starting splunk as root and it starts fine with an alternate port.

Sorry, I don't know what else would cause this issue. Time to engage support? Let us know the cause if you find out!

Re: Splunk instances will not connect to HTTP port

jonesnadiam — Tue, 29 Sep 2020 13:37:30 GMT

FYI -

We needed to update SPLUNK_OS_USER=splunk to SPLUNK_OS_USER=root in $SPLUNK_HOME/etc/splunk-launch.conf.

Spunk was installed and running as root but needed to be started as root. The ownership of the $SPLUNK_HOME directory also needed to be changed to root (instead of splunk). Changing the line above solved the problem.

Thanks so much for the help suarezry! 🙂

Re: Splunk instances will not connect to HTTP port

jonesnadiam — Sun, 09 Apr 2017 22:49:55 GMT

Answered above - thanks again for the help!

Re: Splunk instances will not connect to HTTP port

anand_singh17 — Sun, 13 Aug 2017 05:47:58 GMT

in case of non-root user, what should be the option?

Re: Splunk instances will not connect to HTTP port

anand_singh17 — Sun, 13 Aug 2017 05:50:10 GMT

you need to check you bucket status. check your splunkd.log.

you will get the actual reason for it.

Re: Splunk instances will not connect to HTTP port

mpavlas — Mon, 13 Dec 2021 09:39:16 GMT

More secure way:
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunkd
echo /opt/splunk/lib | sudo tee /etc/ld.so.conf.d/splunk.conf
sudo ldconfig
Then you can run Splunk as non-root user on port 443.

Re: Splunk instances will not connect to HTTP port

mpavlas — Mon, 13 Dec 2021 10:00:07 GMT

Sorry, you need
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunk
as well