Splunk Enterprise 9.3.2 Universal Forwarder node problems

gelfandbein — Thu, 21 Nov 2024 16:51:41 GMT

We try to setup Splunk Enterprise 9.3.2 cluster

All nodes working fine but Splunk Universal Forwarder isn't working - not listening Management port 8089 or 8088...

Running on Google Cloud Platform using RHEL 9.5 (latest) already tried RHEL 8.10 (latest) too

Used documentation: https://docs.splunk.com/Documentation/Forwarder/9.3.2/Forwarder/Installanixuniversalforwarder#Install_the_universal_forwarder_on_Linux

using next commands to setup:

cd /opt

tar xzf /opt/splunkforwarder-9.3.2-d8bb32809498-Linux-x86_64.tgz

adduser -d /opt/splunkforwarder splunkfwd

export SPLUNK_HOME=/opt/splunkforwarder

$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user splunkfwd -group splunkfwd

systemctl start SplunkForwarder

cat /etc/systemd/system/SplunkForwarder.service

[Unit]

Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'

After=network-online.target

Wants=network-online.target

[Service]

Type=simple

Restart=always

ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd --accept-license

KillMode=mixed

KillSignal=SIGINT

TimeoutStopSec=360

LimitNOFILE=65536

LimitRTPRIO=99

SuccessExitStatus=51 52

RestartPreventExitStatus=51

RestartForceExitStatus=52

User=splunkfwd

Group=splunkfwd

NoNewPrivileges=yes

PermissionsStartOnly=true

AmbientCapabilities=CAP_DAC_READ_SEARCH

ExecStartPre=-/bin/bash -c "chown -R splunkfwd:splunkfwd /opt/splunkforwarder"

---

$ cat /etc/os-release

NAME="Red Hat Enterprise Linux"

VERSION="9.5 (Plow)"

ID="rhel"

ID_LIKE="fedora"

VERSION_ID="9.5"

PLATFORM_ID="platform:el9"

PRETTY_NAME="Red Hat Enterprise Linux 9.5 (Plow)"

ANSI_COLOR="0;31"

LOGO="fedora-logo-icon"

CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"

HOME_URL="https://www.redhat.com/"

DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"

BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"

REDHAT_BUGZILLA_PRODUCT_VERSION=9.5

REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"

REDHAT_SUPPORT_PRODUCT_VERSION="9.5"

---

$ netstat -tulpn

[root@splunk-custom-image log]# netstat -tulpn

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1684/sshd: /usr/sbi

tcp6 0 0 :::22 :::* LISTEN 1684/sshd: /usr/sbi

tcp6 0 0 :::20201 :::* LISTEN 2517/otelopscol

udp 0 0 127.0.0.1:323 0.0.0.0:* 652/chronyd

udp6 0 0 ::1:323 :::* 652/chronyd

---

/var/log/messages:

[root@splunk-custom-image log]# systemctl status SplunkForwarder

● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'

Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; preset: disabled)

Active: active (running) since Thu 2024-11-21 09:03:55 EST; 7min ago

Process: 797 ExecStartPre=/bin/bash -c chown -R splunkfwd:splunkfwd /opt/splunkforwarder (code=exited, status=0/SUCCESS)

Main PID: 1068 (splunkd)

Tasks: 47 (limit: 100424)

Memory: 227.4M

CPU: 3.481s

CGroup: /system.slice/SplunkForwarder.service

├─1068 splunkd --under-systemd --systemd-delegate=no -p 8089 _internal_launch_under_systemd

└─2535 "[splunkd pid=1068] splunkd --under-systemd --systemd-delegate=no -p 8089 _internal_launch_under_systemd [process-runner]"

Nov 21 09:03:55 systemd[1]: Started Systemd service file for Splunk, generated by 'splunk enable boot-start'.

Nov 21 09:03:58 splunk[1068]: Warning: Attempting to revert the SPLUNK_HOME ownership

Nov 21 09:03:58 splunk[1068]: Warning: Executing "chown -R splunkfwd:splunkfwd /opt/splunkforwarder"

Nov 21 09:03:58 splunk[1068]: Checking mgmt port [8089]: open

Nov 21 09:03:59 splunk[1068]: Checking conf files for problems...

Nov 21 09:03:59 splunk[1068]: Done

Nov 21 09:03:59 splunk[1068]: Checking default conf files for edits...

Nov 21 09:03:59 splunk[1068]: Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.3.2-d8bb32809498-linux-2.6-x86_64->

Nov 21 09:04:00 splunk[1068]: PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped>

Nov 21 09:04:00 splunk[1068]: 2024-11-21 09:04:00.038 -0500 splunkd started (build d8bb32809498) pid=1068

---

/opt/splunkforwarder/var/log/splunk/splunkd.log

attached file

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems

richgalloway — Thu, 21 Nov 2024 18:17:02 GMT

For security, Splunk UFs default to not listening on a management port. You must explicitly enable it.

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems

gelfandbein — Fri, 22 Nov 2024 12:21:26 GMT

Thanks. But I research documentation how to enable HEC from configuration files - no results. And do not find any link how to enable management port. Maybe you can help with direct link?

$cat /opt/splunkforwarder/etc/apps/splunk_httpinput/local/inputs.conf:

[http] disabled = 0

$cat /opt/splunkforwarder/etc/system/local/inputs.conf:

[http] disabled = 0 [http://input] disabled = 0

Used: https://docs.splunk.com/Documentation/Splunk/9.3.2/Data/UseHECusingconffiles

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems

richgalloway — Fri, 22 Nov 2024 13:19:01 GMT

There is the management mode setting that controls whether the UF listens to a TCP port or via UDS. See https://docs.splunk.com/Documentation/Forwarder/9.3.2/Forwarder/AboutManagementMode

The management port itself is set in web.conf, not inputs .conf (it's not a data input).

[settings] mgmtHostPort = 127.0.0.1:9089

UFs do not support HTTP input.

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems

gelfandbein — Fri, 22 Nov 2024 13:43:13 GMT

@richgalloway Hi there. Thanks for the answer about MGMT port.

I little confusing your answer about that UF do not support HEC.

Previous version 8.2.6 of UF does working fine as HEC with binded 8088 port and forward through TCP data to Indexer nodes (9997) .

Maybe Splunk removed it logic from UF in next versions after 8.2.6?

What is replacement for HEC?

We using UF because parsing do not using license.

What is latest version of UF that can be configured as HTTP Event Collector?

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems

richgalloway — Fri, 22 Nov 2024 15:16:33 GMT

At one time, only indexers and HFs could accept HTTP input. I do not see that documented anywhere now, however.

UFs do very little parsing, except for INDEXED_EXTRACTIONs.

topic Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems in Splunk Enterprise

Splunk Enterprise 9.3.2 Universal Forwarder node problems

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems

Re: Splunk Enterprise 9.3.2 Universal Forwarder node problems