Splunk Enterprise 9.3.2 Universal Forwarder node problems

[gelfandbein]

We try to setup Splunk Enterprise 9.3.2 cluster

All nodes working fine but Splunk Universal Forwarder isn't working - not listening Management port 8089 or 8088...

Running on Google Cloud Platform using RHEL 9.5 (latest) already tried RHEL 8.10 (latest) too

Used documentation: https://docs.splunk.com/Documentation/Forwarder/9.3.2/Forwarder/Installanixuniversalforwarder#Instal...

using next commands to setup:

cd /opt

tar xzf /opt/splunkforwarder-9.3.2-d8bb32809498-Linux-x86_64.tgz

adduser -d /opt/splunkforwarder splunkfwd

export SPLUNK_HOME=/opt/splunkforwarder

$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user splunkfwd -group splunkfwd

systemctl start SplunkForwarder

cat /etc/systemd/system/SplunkForwarder.service

[Unit]

Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'

After=network-online.target

Wants=network-online.target

[Service]

Type=simple

Restart=always

ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd --accept-license

KillMode=mixed

KillSignal=SIGINT

TimeoutStopSec=360

LimitNOFILE=65536

LimitRTPRIO=99

SuccessExitStatus=51 52

RestartPreventExitStatus=51

RestartForceExitStatus=52

User=splunkfwd

Group=splunkfwd

NoNewPrivileges=yes

PermissionsStartOnly=true

AmbientCapabilities=CAP_DAC_READ_SEARCH

ExecStartPre=-/bin/bash -c "chown -R splunkfwd:splunkfwd /opt/splunkforwarder"

---

$ cat /etc/os-release

NAME="Red Hat Enterprise Linux"

VERSION="9.5 (Plow)"

ID="rhel"

ID_LIKE="fedora"

VERSION_ID="9.5"

PLATFORM_ID="platform:el9"

PRETTY_NAME="Red Hat Enterprise Linux 9.5 (Plow)"

ANSI_COLOR="0;31"

LOGO="fedora-logo-icon"

CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"

HOME_URL="https://www.redhat.com/"

DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"

BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"

REDHAT_BUGZILLA_PRODUCT_VERSION=9.5

REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"

REDHAT_SUPPORT_PRODUCT_VERSION="9.5"

---

$ netstat -tulpn

[root@splunk-custom-image log]# netstat -tulpn

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1684/sshd: /usr/sbi

tcp6       0      0 :::22                   :::*                    LISTEN      1684/sshd: /usr/sbi

tcp6       0      0 :::20201                :::*                    LISTEN      2517/otelopscol

udp        0      0 127.0.0.1:323           0.0.0.0:*                           652/chronyd

udp6       0      0 ::1:323                 :::*                                652/chronyd

---

/var/log/messages:

[root@splunk-custom-image log]# systemctl status SplunkForwarder

● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'

     Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; preset: disabled)

     Active: active (running) since Thu 2024-11-21 09:03:55 EST; 7min ago

    Process: 797 ExecStartPre=/bin/bash -c chown -R splunkfwd:splunkfwd /opt/splunkforwarder (code=exited, status=0/SUCCESS)

   Main PID: 1068 (splunkd)

      Tasks: 47 (limit: 100424)

     Memory: 227.4M

        CPU: 3.481s

     CGroup: /system.slice/SplunkForwarder.service

             ├─1068 splunkd --under-systemd --systemd-delegate=no -p 8089 _internal_launch_under_systemd

             └─2535 "[splunkd pid=1068] splunkd --under-systemd --systemd-delegate=no -p 8089 _internal_launch_under_systemd [process-runner]"

Nov 21 09:03:55 systemd[1]: Started Systemd service file for Splunk, generated by 'splunk enable boot-start'.

Nov 21 09:03:58 splunk[1068]: Warning: Attempting to revert the SPLUNK_HOME ownership

Nov 21 09:03:58 splunk[1068]: Warning: Executing "chown -R splunkfwd:splunkfwd /opt/splunkforwarder"

Nov 21 09:03:58 splunk[1068]:         Checking mgmt port [8089]: open

Nov 21 09:03:59 splunk[1068]:         Checking conf files for problems...

Nov 21 09:03:59 splunk[1068]:         Done

Nov 21 09:03:59 splunk[1068]:         Checking default conf files for edits...

Nov 21 09:03:59 splunk[1068]:         Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.3.2-d8bb32809498-linux-2.6-x86_64->

Nov 21 09:04:00 splunk[1068]: PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped>

Nov 21 09:04:00 splunk[1068]: 2024-11-21 09:04:00.038 -0500 splunkd started (build d8bb32809498) pid=1068

---

/opt/splunkforwarder/var/log/splunk/splunkd.log

• attached file

[richgalloway]

There is the management mode setting that controls whether the UF listens to a TCP port or via UDS.  See https://docs.splunk.com/Documentation/Forwarder/9.3.2/Forwarder/AboutManagementMode

The management port itself is set in web.conf, not inputs .conf (it's not a data input).

[settings]
mgmtHostPort = 127.0.0.1:9089

UFs do not support HTTP input.

[gelfandbein]

@richgalloway Hi there. Thanks for the answer about MGMT port.

I little confusing your answer about that UF do not support HEC.

Previous version 8.2.6 of UF does working fine as HEC with binded 8088 port and forward through TCP data to Indexer nodes (9997) .

Maybe Splunk removed it logic from UF in next versions after 8.2.6?

What is replacement for HEC?

We using UF because parsing do not using license.

What is latest version of UF that can be configured as HTTP Event Collector?

[richgalloway]

At one time, only indexers and HFs could accept HTTP input.  I do not see that documented anywhere now, however.

UFs do very little parsing, except for INDEXED_EXTRACTIONs.

[gelfandbein]

Thanks. But I research documentation how to enable HEC from configuration files - no results. And do not find any link how to enable management port. Maybe you can help with direct link?

$cat /opt/splunkforwarder/etc/apps/splunk_httpinput/local/inputs.conf:

[http]
disabled = 0

$cat /opt/splunkforwarder/etc/system/local/inputs.conf:

[http]
disabled = 0

[http://input]
disabled = 0

Used: https://docs.splunk.com/Documentation/Splunk/9.3.2/Data/UseHECusingconffiles

[richgalloway]

For security, Splunk UFs default to not listening on a management port.  You must explicitly enable it.