Splunk 9.3+ Deployment on CIS Red Hat 9 level 1 image - No GUI

[jwestbank]

Hey Splunk Community,

I was wondering if anyone has figured out what is the cause for the GUI not to work at all in a new install of Splunk 9.3 or 9.4 on a [CIS Red Hat ver. 9 Level 1] image. I have been trying to manage the the Splunk server with the GUI and it just wont come up. I can SSH all day long, but no GUI. I did come to the conclusion that its only on the [CIS Red Hat 9 level 1] image and not on an original RHEL Red Hat 9 image. This issues does not appear on [CIS Red Hat 8 level 1] image. 

If anyone has the fix action to what CIS control configuration is causing this it would be greatly appreciated. I am positive if anyone in the [Gov. sector] is going to be hardening there server with CIS RHEL 9 control images they are going to run across this problem.

Thanks - Johnny

[gcusello]

Hi @jwestbank ,

a very stupid question: did you disabled iptables or firewalld on the port 8000?

Ciao.

Giuseppe

[mikelanghorst]

unnecessary comment.  expect better from a Trust member

[isoutamo]

Do you want to explain what you are meaning?

[jwestbank]

gcusello

Firewalld is enabled and I have all the respective ports enabled as well.

firewall-cmd --zone=public --permanent --add-port 8000/tcp
firewall-cmd --reload

I have worked with Splunk Support and Red Hat Support and they have verified my configuration and still didn't figure it out. So the only thing it could be is a hardening configuration from CIS level 1

Thank you buddy for your polite comments.

[isoutamo]

Has splunk started web gui process and are it listening on that host or is it totally down?

[jwestbank]

Web.conf file configuration

[settings]
enableSplunkWebSSL = true
httpport = 8000

##############################

root@SplunkPROD bin]# ./splunk enable web-ssl
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Your session is invalid. Please login.
Splunk username: jwestbank
Password:
You need to restart the Splunk Server (splunkd) for your changes to take effect.
[root@SplunkPROD bin]# ./splunk restart
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
... [ OK ]
Stopping splunk helpers...
[ OK ]
Done.

Splunk> Now with more code!

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _configtracker _dsappevent _dsclient _dsphonehome _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...

Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-9.3.0-51ccf43db5bd-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done
[ OK ]

Waiting for web server at https://127.0.0.1:8000 to be available............WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
. Done

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at https://SplunkPROD:8000

[jwestbank]

root@SplunkPROD bin]# ss -tunlp | grep 8000
tcp LISTEN 0 128 0.0.0.0:8000 0.0.0.0:* users:(("splunkd",pid=17948,fd=179))
[root@SplunPROD bin]#

root@SplunkPROD bin]# firewall-cmd --list-ports
443/tcp 8000/tcp 8089/tcp 8191/tcp 9997/tcp 8000/udp 9997/udp
[root@SplunkPROD bin]#

root@SplunkPROD bin]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: cockpit dhcpv6-client ssh
ports: 8000/tcp 8000/udp 8089/tcp 8191/tcp 443/tcp 9997/tcp 9997/udp
protocols:
forward: yes
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
[root@SplunkPROD bin]#

[isoutamo]

Based on that, it’s up and running.

Have you try it with that node e.g. with curl and using localhost?

How about splunkd_acces.log and web logs etc? Are there anything?

And selinux? Any entries on auditd?

[jwestbank]

Yes I did these commands many times it connects. It was puzzling the Splunk guys and I didn't find anything in the logs and  I sent the Splunk-Diag to the Splunk engineers and they found nothing.

I think we need a Red Hat expert who can look at the CIS hardening controls and state its that one. 

Splunk Leadership should definitely step in and find a solution if this is a bug with CIS Red Hat 9 " v2 " level 1, with there Splunk product 9.3 and 9.4 application.

[isoutamo]

If you can connect locally with curl everything is basically ok. It means that issue is on network side. Have you any node on the same subnet (no network fw between it and splunk), where you could try curl to this host?

Another test which needs to do is try curl on splunk host, but use the official url not localhost. And if there are LB/VIP address before splunk nodes, then use also that and splunk nodes ip too.

In that way we can try to find where the blocking fw.

We have several RHEL 9 cis v1 hardened boxes and there is no issues with them.