Security

Splunk instances will not connect to HTTP port

jonesnadiam
Path Finder

After installing SSL certificates and changing the default Splunk web port to 443, I receive the following error:

Checking http port [443]: already bound
ERROR: The http port [443] is already bound. Splunk needs to use this port.

After killing the processes associated with this port and rebooting, I am still unable to start Splunk, receiving the same error. I've also tried the following with no luck:

  • Confirmed the loopback address in /etc/hosts and ifconfig lo
  • Confirmed there was no BIND_IP defined in /opt/splunk/etc/splunk-launch.conf
  • Tried removing/re-installing Splunk

Any other suggestions?

Note - I have 5 instances of Splunk (1DS, 2SH, 2HF). The only instance that was able to successfully connect to the port was the DS.

1 Solution

jonesnadiam
Path Finder

FYI -

We needed to update SPLUNK_OS_USER=splunk to SPLUNK_OS_USER=root in $SPLUNK_HOME/etc/splunk-launch.conf.

Spunk was installed and running as root but needed to be started as root. The ownership of the $SPLUNK_HOME directory also needed to be changed to root (instead of splunk). Changing the line above solved the problem.

Thanks so much for the help suarezry! 🙂

View solution in original post

mpavlas
Explorer

More secure way:
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunkd
echo /opt/splunk/lib | sudo tee /etc/ld.so.conf.d/splunk.conf
sudo ldconfig
Then you can run Splunk as non-root user on port 443.

0 Karma

mpavlas
Explorer

Sorry, you need
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunk
as well

0 Karma

jonesnadiam
Path Finder

FYI -

We needed to update SPLUNK_OS_USER=splunk to SPLUNK_OS_USER=root in $SPLUNK_HOME/etc/splunk-launch.conf.

Spunk was installed and running as root but needed to be started as root. The ownership of the $SPLUNK_HOME directory also needed to be changed to root (instead of splunk). Changing the line above solved the problem.

Thanks so much for the help suarezry! 🙂

anand_singh17
Path Finder

in case of non-root user, what should be the option?

0 Karma

suarezry
Builder

Is the host windows or linux? From the host try 'telnet localhost 443' to see if that port is in use.

0 Karma

jonesnadiam
Path Finder

This is a Linux server. I receiving the following:

Trying ::1...
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

0 Karma

suarezry
Builder

Ok, so port 443 is free. What user are you trying to run splunk as? Port 443 is a privileged port.

0 Karma

jonesnadiam
Path Finder

Yep - I am running as root 😞

0 Karma

suarezry
Builder

post your $SPLUNK_HOME/etc/system/local/web.conf and server.conf

0 Karma

jonesnadiam
Path Finder

web.conf:

[settings]
httpport = 443
enableSplunkWebSSL = true
privKeyPath =
serverCert =

server.conf:
[general]
serverName = .domain.com

[sslConfig]
sslPassword =

0 Karma

suarezry
Builder

Try changing the port to 8443 just to confirm it starts up fine and not a config issue.

0 Karma

jonesnadiam
Path Finder

Yep, that works fine:

Checking prerequisites...
Checking http port [8443]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
[...]
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

Waiting for web server at https://127.0.0.1:8443 to be available..........

0 Karma

anand_singh17
Path Finder

you need to check you bucket status. check your splunkd.log.

you will get the actual reason for it.

0 Karma

suarezry
Builder

What is the output of these 2 commands:

netstat -na|grep 443
lsof -i|grep 443
0 Karma

jonesnadiam
Path Finder

When running the netstat command, I get the following:
tcp 0 1 :55742 :443 SYN_SENT

When running the lsof command, I get the following:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
connector 3899 root 6u IPv4 92461 0t0 TCP servername.domain.com:58607->name.domain.com:https (SYN_SENT)

0 Karma

suarezry
Builder

Wow ok...So you confirmed nothing is bound to port 443, your loopback and splunk-launch.conf is good, and you are starting splunk as root and it starts fine with an alternate port.

Sorry, I don't know what else would cause this issue. Time to engage support? Let us know the cause if you find out!

0 Karma

jonesnadiam
Path Finder

Answered above - thanks again for the help!

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...