Splunk instances will not connect to HTTP port

[jonesnadiam]

After installing SSL certificates and changing the default Splunk web port to 443, I receive the following error:

Checking http port [443]: already bound
ERROR: The http port [443] is already bound. Splunk needs to use this port.

After killing the processes associated with this port and rebooting, I am still unable to start Splunk, receiving the same error. I've also tried the following with no luck:

• Confirmed the loopback address in /etc/hosts and ifconfig lo
• Confirmed there was no BIND_IP defined in /opt/splunk/etc/splunk-launch.conf
• Tried removing/re-installing Splunk

Any other suggestions?

Note - I have 5 instances of Splunk (1DS, 2SH, 2HF). The only instance that was able to successfully connect to the port was the DS.

[mpavlas]

More secure way:
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunkd
echo /opt/splunk/lib | sudo tee /etc/ld.so.conf.d/splunk.conf
sudo ldconfig
Then you can run Splunk as non-root user on port 443.

[mpavlas]

Sorry, you need
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunk
as well

[jonesnadiam · Accepted solution]

FYI -

We needed to update SPLUNK_OS_USER=splunk to SPLUNK_OS_USER=root in $SPLUNK_HOME/etc/splunk-launch.conf.

Spunk was installed and running as root but needed to be started as root. The ownership of the $SPLUNK_HOME directory also needed to be changed to root (instead of splunk). Changing the line above solved the problem.

Thanks so much for the help suarezry! 🙂

[anand_singh17]

in case of non-root user, what should be the option?

[suarezry]

Is the host windows or linux? From the host try 'telnet localhost 443' to see if that port is in use.

[jonesnadiam]

This is a Linux server. I receiving the following:

Trying ::1...
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

[suarezry]

Ok, so port 443 is free. What user are you trying to run splunk as? Port 443 is a privileged port.

[jonesnadiam]

Yep - I am running as root 😞

[suarezry]

post your $SPLUNK_HOME/etc/system/local/web.conf and server.conf

[jonesnadiam]

web.conf:

[settings]
httpport = 443
enableSplunkWebSSL = true
privKeyPath =
serverCert =

server.conf:
[general]
serverName = .domain.com

[sslConfig]
sslPassword =

[suarezry]

Try changing the port to 8443 just to confirm it starts up fine and not a config issue.

[jonesnadiam]

Yep, that works fine:

Checking prerequisites...
Checking http port [8443]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
[...]
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

Waiting for web server at https://127.0.0.1:8443 to be available..........

[anand_singh17]

you need to check you bucket status. check your splunkd.log.

you will get the actual reason for it.

[suarezry]

What is the output of these 2 commands:

netstat -na|grep 443
lsof -i|grep 443

[jonesnadiam]

When running the netstat command, I get the following:
tcp 0 1 :55742 :443 SYN_SENT

When running the lsof command, I get the following:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
connector 3899 root 6u IPv4 92461 0t0 TCP servername.domain.com:58607->name.domain.com:https (SYN_SENT)

[suarezry]

Wow ok...So you confirmed nothing is bound to port 443, your loopback and splunk-launch.conf is good, and you are starting splunk as root and it starts fine with an alternate port.

Sorry, I don't know what else would cause this issue. Time to engage support? Let us know the cause if you find out!

[jonesnadiam]

Answered above - thanks again for the help!