systemctl start SplunkForwarder fails error=203

[allroadsleadtoa]

got an alert that splunk is not running. Tried to restart using systemd restart SplunkForwarder.

● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Mon 2020-02-24 07:25:40 MST; 1 day 1h ago
Process: 344227 ExecStartPost=/bin/bash -c chown -R 2080:2080 /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=
Process: 344225 ExecStartPost=/bin/bash -c chown -R 2080:2080 /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/S
Process: 344224 ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=203/EXEC)
Main PID: 344224 (code=exited, status=203/EXEC)

Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enab
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Unit SplunkForwarder.service entered failed state.
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: SplunkForwarder.service failed.
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: SplunkForwarder.service holdoff time over, scheduling restart.
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: start request repeated too quickly for SplunkForwarder.service
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enab
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Unit SplunkForwarder.service entered failed state.
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: SplunkForwarder.service failed.

[codebuilder]

Make sure that all files and directories under $SPLUNK_HOME are owned by splunk, or whatever user you chose, and not owned by root.

[garias_splunk]

I had exactly the same issue on RHEL8 and the problem was SELinux blocking this service. I had:

# getenforce
Enforced

I changed that with this command

# sudo setenforce 0

Once I had that set to Permissive, the service started fine.

# getenforce
Permissive

These were my logs:

[root@Server12345 d3569346]# systemctl status Splunkd.service
● Splunkd.service
Loaded: loaded (/etc/systemd/system/Splunkd.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2020-12-11 16:11:22 HKT; 13s ago
Process: 167388 ExecStartPost=/bin/bash -c chown -R splunk:users /sys/fs/cgroup/memory/system.slice/Splunkd.service (code=exited, status=0/SUCCESS)
Process: 167386 ExecStartPost=/bin/bash -c chown -R splunk:users /sys/fs/cgroup/cpu/system.slice/Splunkd.service (code=exited, status=0/SUCCESS)
Process: 167385 ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd (code=exited, status=203/EXEC)
Main PID: 167385 (code=exited, status=203/EXEC)

Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Failed with result 'exit-code'.
Dec 11 16:11:22 Server12345 systemd[1]: Failed to start Splunkd.service.
Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Service RestartSec=100ms expired, scheduling restart.
Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Scheduled restart job, restart counter is at 5.
Dec 11 16:11:22 Server12345 systemd[1]: Stopped Splunkd.service.
Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Start request repeated too quickly.
Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Failed with result 'exit-code'.
Dec 11 16:11:22 Server12345 systemd[1]: Failed to start Splunkd.service.

*******************************

-- Unit tsSplunk.service has begun starting up.
Dec 21 17:12:30 Server12345 systemd[32167]: tsSplunk.service: Failed to execute command: Permission denied
Dec 21 17:12:30 Server12345 systemd[32167]: tsSplunk.service: Failed at step EXEC spawning /opt/splunk/bin/splunk: Permission denied
-- Subject: Process /opt/splunk/bin/splunk could not be executed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The process /opt/splunk/bin/splunk could not be executed and failed.
--
-- The error number returned by this process is 13.
Dec 21 17:12:30 Server12345 systemd[1]: tsSplunk.service: Main process exited, code=exited, status=203/EXEC
Dec 21 17:12:30 Server12345 systemd[1]: tsSplunk.service: Failed with result 'exit-code'.
Dec 21 17:12:30 Server12345 systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
-- Subject: Unit tsSplunk.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support

[adamsaul]

What UF version is this?

Recently, Splunk switched over to making the UFs register as splunk. That way the systemd name is same between a Splunk "full" install or UF.

Try this command to see what it is registered:
systemctl -l | grep -i splunk

[ephemeric]

On CentOS 7.9:

$> systemctl list-unit-files | grep -i splunk
splunkforwarder.service enabled

 Package:

splunkforwarder-8.2.1-ddff1c41e5cf-linux-2.6-x86_64.rpm