- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We are unable to access our splunk webserver post OS upgrade to RHEL 8.10, However our Splunk service is up and running fine but the UI is not available for us. Can someone please help us to fix this issue.
We have checked port 8000 is listening fine.
Trying 10.xxx.xxx.xx...
Connected to 10.xxx.xxx.xx.
Escape character is '^]'.
^Z
Connection closed by foreign host.
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN
We dont have anything in internal logs to dig in.
we are getting the error message on the page as below:
The connection has timed out
The server at 10.xxx.xxx.xx is taking too long to respond.
- The site could be temporarily unavailable or too busy. Try again in a few moments.
- If you are unable to load any pages, check your computer’s network connection.
- If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the web.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Internally on the server, Splunk is running fine. Externally, if you cannot access http://<server-ip>:8000, the firewall may be blocking the connection. Contact your OS team to fix this.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When i ran the command its shows nothing, does that means 8000 port is not open?
sudo firewall-cmd --list-ports
[acnops_splunk@IEM***** ~]$ netstat -tulnp | grep 8000
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 3679909/splunkd
[acnops_splunk@IEMS****** ~]$
netstat -tulnp | grep 8000
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Internally on the server, Splunk is running fine. Externally, if you cannot access http://<server-ip>:8000, the firewall may be blocking the connection. Contact your OS team to fix this.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It seems that Splunk Web (port 8000) is not open in the firewall. Please coordinate with the firewall or OS team and inform them about this. We recently encountered a similar issue, and after notifying the Linux team, they made some firewall-level changes at the OS level. Following those changes, we were able to access the Splunk Web successfully.
sudo firewall-cmd --permanent --add-port=8000/tcp
sudo firewall-cmd --reload
Check if the port is open.
sudo firewall-cmd --list-ports
If you see 8000/tcp in the list, the port is now open.
After opening the port, restart Splunk
sudo su - splunk
/opt/splunk/bin/splunk restart
Check if Splunk Web is listening on port 8000:
netstat -tulnp | grep 8000
If firewalld is disabled but iptables is in use, you need to allow port 8000:
sudo iptables -A INPUT -p tcp --dport 8000 -j ACCEPT
sudo iptables-save | sudo tee /etc/sysconfig/iptables
sudo systemctl restart iptables
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Check if SELinux is Blocking Access. After upgrading to RHEL 8.10, SELinux policies may restrict Splunk Web.
Temporarily disable SELinux and test: sudo setenforce 0
If this resolves the issue, permanently disable SELinux by modifying the configuration and Reboot the server
sudo vi /etc/selinux/config
Change: SELINUX=enforcing → SELINUX=disabled
Try accessing the Splunk UI locally from the server to confirm if the firewall is blocking external acces
curl -v <http://splunkipadd>:8000
Check Splunk logs for any web service issues: cat /opt/splunk/var/log/splunk/web_service.log
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @kiran_panchavat ,
Thanks for your response,
However, i checked and found that SElinux is already permissive(0) here
[acnops_splunk@IEM***** ~]$ getenforce
Permissive
Also, i did curl from local server and don't find any connection error showing in the below output
[acnops_splunk@IEM****** ~]$ curl -v http://<serverip>:8000
* Rebuilt URL to: http://<serverip>:8000/
* Trying <serverip>...
* TCP_NODELAY set
* Connected to <serverip> port 8000 (#0)
> GET / HTTP/1.1
> Host: <serverip>:8000
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 303 See Other
< Date: Fri, 07 Feb 2025 13:30:56 GMT
< Content-Type: text/html; charset=UTF-8
< X-Content-Type-Options: nosniff
< Content-Length: 339
< Location: http://<serverip>:8000/en-US/
< Vary: Accept-Language
< Connection: Keep-Alive
< X-Frame-Options: SAMEORIGIN
< Server: Splunkd
<
<!doctype html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta http-equiv="refresh" content="1;url=http://<serverip>:8000/en-US/"><title>303 See Other</title></head><body><h1>See Other</h1><p>The resource has moved temporarily <a href="http://<serverip>/en-US/">here</a>.</p></body></html>
* Connection #0 to host <serverip> left intact
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You need to update the URL: curl -v http://<serverip>:8000
Please replace it with your Splunk Web URL.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, i am doing curl for my web url only and the result is good.
Any further help would really be appreciated.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It looks like port 8000 is already open on the host firewall (I believe "irdmi" referenced is the service name for port 8000 on RHEL), so it sounds like the host itself should be allowing connectivity.
Nevertheless, you could try explicitly allowing port 8000 and checking the logs:
Open Port 8000 in the Firewall:
sudo firewall-cmd --zone=public --add-port=8000/tcp --permanent sudo firewall-cmd --reloadVerify with:
sudo firewall-cmd --list-all- Check Splunk logs for any errors:
$SPLUNK_HOME/var/log/splunk/web_service.log
$SPLUNK_HOME/var/log/splunk/splunkd.log
Have you been able to confirm that no network changes were made around the time?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Check Splunk logs for any web service issues: cat /opt/splunk/var/log/splunk/web_service.log
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When you ran the telnet check, was this from the same host you are trying to access Splunk with via the browser or from the Splunk server itself?
If this was checked from the Splunk server then I would suggest checking the firewall rules on that host if either `iptables` or `firewalld` is configured to allow inbound traffic on port 8000. You can check your firewall rules with: `sudo iptables -L` or `sudo firewall-cmd --list-all` depending how this is configured on your host.
Please check if you are using https in your URL if Splunk has been configured with SSL enabled.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We are using http url with setting enableSplunkWebSSL = false in web.conf file.
The host where i am trying to access splunk webrowser is a windows machine and the telnet i did is from the splunk server that is a linux machine which i am trying to access and its not accessible in url.
below output from splunk server:
sudo iptables -L
[sudo] password for acnops_splunk:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:irdmi
ACCEPT tcp -- anywhere anywhere tcp dpt:palace-6
ACCEPT tcp -- anywhere anywhere tcp dpt:distinct32
ACCEPT tcp -- anywhere anywhere tcp dpt:8089
ACCEPT tcp -- anywhere anywhere tcp dpt:distinct
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[acnops_splunk@IEM******** ~]$ sudo firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: dhcpv6-client ssh
ports:
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
[acnops_splunk@IEM****** ~]$
looking forward for some solution
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Since you apparently did a local connectivity test and it succeeded, there must be something external to Splunk itself preventing you from connecting. Your iptables rules seem to not be interfering (you don't have port 8000 explicitly open but the general policy is ACCEPT). So it points to something network-related. Routing? Filtering on some intermediate device? It's something best solved with your local admin staff since it doesn't seem to be related to Splunk as such.
