We try to setup Splunk Enterprise 9.3.2 cluster All nodes working fine but Splunk Universal Forwarder isn't working - not listening Management port 8089 or 8088... Running on Google Cloud Platform using RHEL 9.5 (latest) already tried RHEL 8.10 (latest) too Used documentation: https://docs.splunk.com/Documentation/Forwarder/9.3.2/Forwarder/Installanixuniversalforwarder#Install_the_universal_forwarder_on_Linux using next commands to setup: cd /opt tar xzf /opt/splunkforwarder-9.3.2-d8bb32809498-Linux-x86_64.tgz adduser -d /opt/splunkforwarder splunkfwd export SPLUNK_HOME=/opt/splunkforwarder $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user splunkfwd -group splunkfwd systemctl start SplunkForwarder cat /etc/systemd/system/SplunkForwarder.service [Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network-online.target Wants=network-online.target [Service] Type=simple Restart=always ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd --accept-license KillMode=mixed KillSignal=SIGINT TimeoutStopSec=360 LimitNOFILE=65536 LimitRTPRIO=99 SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 User=splunkfwd Group=splunkfwd NoNewPrivileges=yes PermissionsStartOnly=true AmbientCapabilities=CAP_DAC_READ_SEARCH ExecStartPre=-/bin/bash -c "chown -R splunkfwd:splunkfwd /opt/splunkforwarder" --- $ cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="9.5 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.5" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.5 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9" BUG_REPORT_URL="https://issues.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.5 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.5" --- $ netstat -tulpn [root@splunk-custom-image log]# netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1684/sshd: /usr/sbi tcp6 0 0 :::22 :::* LISTEN 1684/sshd: /usr/sbi tcp6 0 0 :::20201 :::* LISTEN 2517/otelopscol udp 0 0 127.0.0.1:323 0.0.0.0:* 652/chronyd udp6 0 0 ::1:323 :::* 652/chronyd --- /var/log/messages: [root@splunk-custom-image log]# systemctl status SplunkForwarder ● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start' Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; preset: disabled) Active: active (running) since Thu 2024-11-21 09:03:55 EST; 7min ago Process: 797 ExecStartPre=/bin/bash -c chown -R splunkfwd:splunkfwd /opt/splunkforwarder (code=exited, status=0/SUCCESS) Main PID: 1068 (splunkd) Tasks: 47 (limit: 100424) Memory: 227.4M CPU: 3.481s CGroup: /system.slice/SplunkForwarder.service ├─1068 splunkd --under-systemd --systemd-delegate=no -p 8089 _internal_launch_under_systemd └─2535 "[splunkd pid=1068] splunkd --under-systemd --systemd-delegate=no -p 8089 _internal_launch_under_systemd [process-runner]" Nov 21 09:03:55 systemd[1]: Started Systemd service file for Splunk, generated by 'splunk enable boot-start'. Nov 21 09:03:58 splunk[1068]: Warning: Attempting to revert the SPLUNK_HOME ownership Nov 21 09:03:58 splunk[1068]: Warning: Executing "chown -R splunkfwd:splunkfwd /opt/splunkforwarder" Nov 21 09:03:58 splunk[1068]: Checking mgmt port [8089]: open Nov 21 09:03:59 splunk[1068]: Checking conf files for problems... Nov 21 09:03:59 splunk[1068]: Done Nov 21 09:03:59 splunk[1068]: Checking default conf files for edits... Nov 21 09:03:59 splunk[1068]: Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.3.2-d8bb32809498-linux-2.6-x86_64-> Nov 21 09:04:00 splunk[1068]: PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped> Nov 21 09:04:00 splunk[1068]: 2024-11-21 09:04:00.038 -0500 splunkd started (build d8bb32809498) pid=1068 --- /opt/splunkforwarder/var/log/splunk/splunkd.log attached file
... View more