Building for the Splunk Platform

systemctl start SplunkForwarder fails error=203

allroadsleadtoa
New Member

got an alert that splunk is not running. Tried to restart using systemd restart SplunkForwarder.

● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Mon 2020-02-24 07:25:40 MST; 1 day 1h ago
Process: 344227 ExecStartPost=/bin/bash -c chown -R 2080:2080 /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=
Process: 344225 ExecStartPost=/bin/bash -c chown -R 2080:2080 /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/S
Process: 344224 ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=203/EXEC)
Main PID: 344224 (code=exited, status=203/EXEC)

Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enab
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Unit SplunkForwarder.service entered failed state.
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: SplunkForwarder.service failed.
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: SplunkForwarder.service holdoff time over, scheduling restart.
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: start request repeated too quickly for SplunkForwarder.service
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enab
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: Unit SplunkForwarder.service entered failed state.
Feb 24 07:25:40 pplx2dbadm05.adt.com systemd[1]: SplunkForwarder.service failed.

Tags (1)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

Make sure that all files and directories under $SPLUNK_HOME are owned by splunk, or whatever user you chose, and not owned by root.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

garias_splunk
Splunk Employee
Splunk Employee

I had exactly the same issue on RHEL8 and the problem was SELinux blocking this service. I had:

# getenforce
Enforced

I changed that with this command

# sudo setenforce 0

Once I had that set to Permissive, the service started fine.

# getenforce
Permissive

 

These were my logs:

[root@Server12345 d3569346]# systemctl status Splunkd.service
● Splunkd.service
Loaded: loaded (/etc/systemd/system/Splunkd.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2020-12-11 16:11:22 HKT; 13s ago
Process: 167388 ExecStartPost=/bin/bash -c chown -R splunk:users /sys/fs/cgroup/memory/system.slice/Splunkd.service (code=exited, status=0/SUCCESS)
Process: 167386 ExecStartPost=/bin/bash -c chown -R splunk:users /sys/fs/cgroup/cpu/system.slice/Splunkd.service (code=exited, status=0/SUCCESS)
Process: 167385 ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd (code=exited, status=203/EXEC)
Main PID: 167385 (code=exited, status=203/EXEC)

Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Failed with result 'exit-code'.
Dec 11 16:11:22 Server12345 systemd[1]: Failed to start Splunkd.service.
Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Service RestartSec=100ms expired, scheduling restart.
Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Scheduled restart job, restart counter is at 5.
Dec 11 16:11:22 Server12345 systemd[1]: Stopped Splunkd.service.
Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Start request repeated too quickly.
Dec 11 16:11:22 Server12345 systemd[1]: Splunkd.service: Failed with result 'exit-code'.
Dec 11 16:11:22 Server12345 systemd[1]: Failed to start Splunkd.service.

*******************************

-- Unit tsSplunk.service has begun starting up.
Dec 21 17:12:30 Server12345 systemd[32167]: tsSplunk.service: Failed to execute command: Permission denied
Dec 21 17:12:30 Server12345 systemd[32167]: tsSplunk.service: Failed at step EXEC spawning /opt/splunk/bin/splunk: Permission denied
-- Subject: Process /opt/splunk/bin/splunk could not be executed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The process /opt/splunk/bin/splunk could not be executed and failed.
--
-- The error number returned by this process is 13.
Dec 21 17:12:30 Server12345 systemd[1]: tsSplunk.service: Main process exited, code=exited, status=203/EXEC
Dec 21 17:12:30 Server12345 systemd[1]: tsSplunk.service: Failed with result 'exit-code'.
Dec 21 17:12:30 Server12345 systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
-- Subject: Unit tsSplunk.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support

 

Tags (2)
0 Karma

adamsaul
Communicator

What UF version is this?

Recently, Splunk switched over to making the UFs register as splunk. That way the systemd name is same between a Splunk "full" install or UF.

Try this command to see what it is registered:
systemctl -l | grep -i splunk

0 Karma

ephemeric
Contributor

On CentOS 7.9:

$> systemctl list-unit-files | grep -i splunk
splunkforwarder.service enabled

 Package:

splunkforwarder-8.2.1-ddff1c41e5cf-linux-2.6-x86_64.rpm

 

0 Karma
Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...