Cannot connect remotely to Splunk Web interface

[reswob4]

I have a problem where I cannot remotely access the web interface (not via HTTPS or HTTP on either 8000 or 8089) of one of my Windows HF.

However, when I RDP to the server, I can launch a browser and connect to either the loopback IP (https://127.0.0.1:8000) or the public IP (https://x.x.x.x:8000)

If I check wireshark, I see the syn packets come in, but no response at all being sent out.

This, of course, points to the firewall being a problem, but I checked the firewall and it is configured to accept all connections from any IP on any port for the Splunk application (and the splunk web application, but if I remember correctly, that's not used anymore). I tried disabling the firewall by going to Windows Firewall with Advanced Security on Local Computer Properties and turning the firewall state to OFF, but I still couldn't connect.

This started sometime in the past month. Since I don't routinely connect to this interface, I don't know when it stopped working. And it means it used to work without outgoing firewall rules configured.

There are NO references to failed connections or anything similar in splunkd.log. I enabled the firewall text log, but that genereated no entrieds either (or I'm doing that wrong).

Here's the output from the startup:

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Splunkd: Starting (pid 4312)
Done

Waiting for web server at https://127.0.0.1:8000 to be available... Done

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at https://splunk-04:8000

C:>"Program Files\Splunk\bin"\splunk status
Splunkd: Running (pid 4312)

Here's the output for the ports from netstat

C:>netstat -anob | findstr 8000
TCP 0.0.0.0:8000 0.0.0.0:0 LISTENING 6852

C:>netstat -anob | findstr 8089
TCP 0.0.0.0:8089 0.0.0.0:0 LISTENING 6852

Here are some links to other things I've tried:

https://answers.splunk.com/answers/26598/splunk-receiver-listening-on-9997-but-only-on-localhost.htm...

https://answers.splunk.com/answers/179823/splunk-always-starts-at-1270018000.html

https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/BindSplunktoanIP

neither setting SPLUNK_BINDIP=0.0.0.0 or SPLUNK_BINDIP= works

The only things I know have changed (besides patches being applied) are java 1.8 and Logstash 5.4.1 were installed. Logstash is NOT running.

While I go create some outbound firewall rules to see if that works, I figured I'd post here to see if anyone else has any ideas. I also still have to look in the Windows Event log to see if there are entries there...

Thanks for any suggestions.

[woodcock]

Fixed (swapped).

[woodcock]

Check that server for a rogue web.conf with startwebserver = 0; run this on the CLI:

$SPLUNK_HOME/bin/splunk btool web list --debug

BTW, best practice is to leave the splunkweb off on everything except for your Search Heads and to admin via DS or some other configuration manager.

[reswob4]

FYI, you have list and web reversed. But regardless, my startwebserver = 1 so that's not it.

Since I have other pans burning in the fire, and the HF is working doing what a HF is supposed to do, I think I'll let this go for now and manage via the cli.

Thanks