topic Re: Splunk WebServer not available post OS upgrade to RHEL 8.10 in Installation

Splunk WebServer not available post OS upgrade to RHEL 8.10

rukshar — Thu, 06 Feb 2025 12:57:24 GMT

We are unable to access our splunk webserver post OS upgrade to RHEL 8.10, However our Splunk service is up and running fine but the UI is not available for us. Can someone please help us to fix this issue.

We have checked port 8000 is listening fine.

Trying 10.xxx.xxx.xx...
Connected to 10.xxx.xxx.xx.
Escape character is '^]'.
^Z
Connection closed by foreign host.

tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN

We dont have anything in internal logs to dig in.

we are getting the error message on the page as below:

The connection has timed out

The server at 10.xxx.xxx.xx is taking too long to respond.

The site could be temporarily unavailable or too busy. Try again in a few moments.
If you are unable to load any pages, check your computer’s network connection.
If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the web.

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

livehybrid — Thu, 06 Feb 2025 16:00:56 GMT

When you ran the telnet check, was this from the same host you are trying to access Splunk with via the browser or from the Splunk server itself?

If this was checked from the Splunk server then I would suggest checking the firewall rules on that host if either `iptables` or `firewalld` is configured to allow inbound traffic on port 8000. You can check your firewall rules with: `sudo iptables -L` or `sudo firewall-cmd --list-all` depending how this is configured on your host.

Please check if you are using https in your URL if Splunk has been configured with SSL enabled.

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

rukshar — Fri, 07 Feb 2025 09:34:56 GMT

We are using http url with setting enableSplunkWebSSL = false in web.conf file.

The host where i am trying to access splunk webrowser is a windows machine and the telnet i did is from the splunk server that is a linux machine which i am trying to access and its not accessible in url.

below output from splunk server:

sudo iptables -L
[sudo] password for acnops_splunk:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:irdmi
ACCEPT tcp -- anywhere anywhere tcp dpt:palace-6
ACCEPT tcp -- anywhere anywhere tcp dpt:distinct32
ACCEPT tcp -- anywhere anywhere tcp dpt:8089
ACCEPT tcp -- anywhere anywhere tcp dpt:distinct

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[acnops_splunk@IEM******** ~]$ sudo firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: dhcpv6-client ssh
ports:
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
[acnops_splunk@IEM****** ~]$

looking forward for some solution

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

kiran_panchavat — Fri, 07 Feb 2025 13:16:40 GMT

@rukshar

Check if SELinux is Blocking Access. After upgrading to RHEL 8.10, SELinux policies may restrict Splunk Web.

Temporarily disable SELinux and test: sudo setenforce 0

If this resolves the issue, permanently disable SELinux by modifying the configuration and Reboot the server

sudo vi /etc/selinux/config

Change: SELINUX=enforcing → SELINUX=disabled

Try accessing the Splunk UI locally from the server to confirm if the firewall is blocking external acces

curl -v <http://splunkipadd>:8000

Check Splunk logs for any web service issues: cat /opt/splunk/var/log/splunk/web_service.log

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

rukshar — Fri, 07 Feb 2025 14:02:01 GMT

Hello @kiran_panchavat ,

Thanks for your response,
However, i checked and found that SElinux is already permissive(0) here

[acnops_splunk@IEM***** ~]$ getenforce
Permissive

Also, i did curl from local server and don't find any connection error showing in the below output

[acnops_splunk@IEM****** ~]$ curl -v http://<serverip>:8000
* Rebuilt URL to: http://<serverip>:8000/
* Trying <serverip>...
* TCP_NODELAY set
* Connected to <serverip> port 8000 (#0)
> GET / HTTP/1.1
> Host: <serverip>:8000
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 303 See Other
< Date: Fri, 07 Feb 2025 13:30:56 GMT
< Content-Type: text/html; charset=UTF-8
< X-Content-Type-Options: nosniff
< Content-Length: 339
< Location: http://<serverip>:8000/en-US/
< Vary: Accept-Language
< Connection: Keep-Alive
< X-Frame-Options: SAMEORIGIN
< Server: Splunkd
<
<!doctype html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta http-equiv="refresh" content="1;url=http://<serverip>:8000/en-US/"><title>303 See Other</title></head><body><h1>See Other</h1><p>The resource has moved temporarily <a href="http://<serverip>/en-US/">here</a>.</p></body></html>
* Connection #0 to host <serverip> left intact

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

kiran_panchavat — Fri, 07 Feb 2025 16:55:51 GMT

@rukshar

You need to update the URL: curl -v http://<serverip>:8000

Please replace it with your Splunk Web URL.

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

rukshar — Fri, 07 Feb 2025 17:16:44 GMT

Yes, i am doing curl for my web url only and the result is good.
Any further help would really be appreciated.

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

kiran_panchavat — Fri, 07 Feb 2025 17:21:54 GMT

@rukshar

Check Splunk logs for any web service issues: cat /opt/splunk/var/log/splunk/web_service.log

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

kiran_panchavat — Fri, 07 Feb 2025 17:31:34 GMT

@rukshar

It seems that Splunk Web (port 8000) is not open in the firewall. Please coordinate with the firewall or OS team and inform them about this. We recently encountered a similar issue, and after notifying the Linux team, they made some firewall-level changes at the OS level. Following those changes, we were able to access the Splunk Web successfully.

sudo firewall-cmd --permanent --add-port=8000/tcp 
sudo firewall-cmd --reload

Check if the port is open.

sudo firewall-cmd --list-ports

If you see 8000/tcp in the list, the port is now open.

After opening the port, restart Splunk

sudo su - splunk
/opt/splunk/bin/splunk restart

Check if Splunk Web is listening on port 8000:

 netstat -tulnp | grep 8000

If firewalld is disabled but iptables is in use, you need to allow port 8000:

sudo iptables -A INPUT -p tcp --dport 8000 -j ACCEPT
sudo iptables-save | sudo tee /etc/sysconfig/iptables
sudo systemctl restart iptables

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

rukshar — Fri, 07 Feb 2025 17:47:28 GMT

When i ran the command its shows nothing, does that means 8000 port is not open?

sudo firewall-cmd --list-ports

[acnops_splunk@IEM***** ~]$ netstat -tulnp | grep 8000
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 3679909/splunkd
[acnops_splunk@IEMS****** ~]$

netstat -tulnp | grep 8000

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

kiran_panchavat — Fri, 07 Feb 2025 17:53:17 GMT

@rukshar

Internally on the server, Splunk is running fine. Externally, if you cannot access http://<server-ip>:8000, the firewall may be blocking the connection. Contact your OS team to fix this.

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

livehybrid — Sat, 08 Feb 2025 08:23:28 GMT

It looks like port 8000 is already open on the host firewall (I believe "irdmi" referenced is the service name for port 8000 on RHEL), so it sounds like the host itself should be allowing connectivity.

Nevertheless, you could try explicitly allowing port 8000 and checking the logs:

Open Port 8000 in the Firewall:
sudo firewall-cmd --zone=public --add-port=8000/tcp --permanent sudo firewall-cmd --reload

Verify with:
sudo firewall-cmd --list-all
Check Splunk logs for any errors:
$SPLUNK_HOME/var/log/splunk/web_service.log
$SPLUNK_HOME/var/log/splunk/splunkd.log

Have you been able to confirm that no network changes were made around the time?

Re: Splunk WebServer not available post OS upgrade to RHEL 8.10

PickleRick — Sat, 08 Feb 2025 11:38:03 GMT

Since you apparently did a local connectivity test and it succeeded, there must be something external to Splunk itself preventing you from connecting. Your iptables rules seem to not be interfering (you don't have port 8000 explicitly open but the general policy is ACCEPT). So it points to something network-related. Routing? Filtering on some intermediate device? It's something best solved with your local admin staff since it doesn't seem to be related to Splunk as such.