Security

Splunk instances will not connect to HTTP port

jonesnadiam
Path Finder

After installing SSL certificates and changing the default Splunk web port to 443, I receive the following error:

Checking http port [443]: already bound
ERROR: The http port [443] is already bound. Splunk needs to use this port.

After killing the processes associated with this port and rebooting, I am still unable to start Splunk, receiving the same error. I've also tried the following with no luck:

  • Confirmed the loopback address in /etc/hosts and ifconfig lo
  • Confirmed there was no BIND_IP defined in /opt/splunk/etc/splunk-launch.conf
  • Tried removing/re-installing Splunk

Any other suggestions?

Note - I have 5 instances of Splunk (1DS, 2SH, 2HF). The only instance that was able to successfully connect to the port was the DS.

1 Solution

jonesnadiam
Path Finder

FYI -

We needed to update SPLUNK_OS_USER=splunk to SPLUNK_OS_USER=root in $SPLUNK_HOME/etc/splunk-launch.conf.

Spunk was installed and running as root but needed to be started as root. The ownership of the $SPLUNK_HOME directory also needed to be changed to root (instead of splunk). Changing the line above solved the problem.

Thanks so much for the help suarezry! 🙂

View solution in original post

mpavlas
Explorer

More secure way:
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunkd
echo /opt/splunk/lib | sudo tee /etc/ld.so.conf.d/splunk.conf
sudo ldconfig
Then you can run Splunk as non-root user on port 443.

0 Karma

mpavlas
Explorer

Sorry, you need
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunk
as well

0 Karma

jonesnadiam
Path Finder

FYI -

We needed to update SPLUNK_OS_USER=splunk to SPLUNK_OS_USER=root in $SPLUNK_HOME/etc/splunk-launch.conf.

Spunk was installed and running as root but needed to be started as root. The ownership of the $SPLUNK_HOME directory also needed to be changed to root (instead of splunk). Changing the line above solved the problem.

Thanks so much for the help suarezry! 🙂

anand_singh17
Path Finder

in case of non-root user, what should be the option?

0 Karma

suarezry
Builder

Is the host windows or linux? From the host try 'telnet localhost 443' to see if that port is in use.

0 Karma

jonesnadiam
Path Finder

This is a Linux server. I receiving the following:

Trying ::1...
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

0 Karma

suarezry
Builder

Ok, so port 443 is free. What user are you trying to run splunk as? Port 443 is a privileged port.

0 Karma

jonesnadiam
Path Finder

Yep - I am running as root 😞

0 Karma

suarezry
Builder

post your $SPLUNK_HOME/etc/system/local/web.conf and server.conf

0 Karma

jonesnadiam
Path Finder

web.conf:

[settings]
httpport = 443
enableSplunkWebSSL = true
privKeyPath =
serverCert =

server.conf:
[general]
serverName = .domain.com

[sslConfig]
sslPassword =

0 Karma

suarezry
Builder

Try changing the port to 8443 just to confirm it starts up fine and not a config issue.

0 Karma

jonesnadiam
Path Finder

Yep, that works fine:

Checking prerequisites...
Checking http port [8443]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
[...]
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

Waiting for web server at https://127.0.0.1:8443 to be available..........

0 Karma

anand_singh17
Path Finder

you need to check you bucket status. check your splunkd.log.

you will get the actual reason for it.

0 Karma

suarezry
Builder

What is the output of these 2 commands:

netstat -na|grep 443
lsof -i|grep 443
0 Karma

jonesnadiam
Path Finder

When running the netstat command, I get the following:
tcp 0 1 :55742 :443 SYN_SENT

When running the lsof command, I get the following:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
connector 3899 root 6u IPv4 92461 0t0 TCP servername.domain.com:58607->name.domain.com:https (SYN_SENT)

0 Karma

suarezry
Builder

Wow ok...So you confirmed nothing is bound to port 443, your loopback and splunk-launch.conf is good, and you are starting splunk as root and it starts fine with an alternate port.

Sorry, I don't know what else would cause this issue. Time to engage support? Let us know the cause if you find out!

0 Karma

jonesnadiam
Path Finder

Answered above - thanks again for the help!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...