Linux UF boot-start issues and non-impacting errors on 9.0.0

_joe — Tue, 21 Jun 2022 13:11:28 GMT

Is anyone else running into boot-start/permissions issues with the 9.0.0 UF running on Linux using init.d scripts for bootstart?

Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder"

I am also finding that "./splunk disable boot-start" does not correctly remove the /etc/init.d/splunk script and, contrary to documentation, splunk UF 9.0.0 uses systemd as default.
https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/ConfigureSplunktostartatboottime

Also systemd scripts seem to fail getting the permissions needed even when trying to enable-boot as root. A key error I am seeing is "Failed to create the unit file" when running the install. But it seems to be a total fail.

## When upgrading (from 8.2.5) runuser -l splunk -c "/opt/splunkforwarder/bin/splunk stop" tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt chown -R splunk:splunk /opt/splunkforwarder/ runuser -l splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt" runuser -l splunk -c "/opt/splunkforwarder/bin/splunk status" Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" (NOTE: Seems to be non-impacting)

### When doing a new install tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt chown -R splunk:splunk /opt/splunkforwarder [root]# sudo -H -u splunk /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" This appears to be your first time running this version of Splunk. IMPORTANT: Because an admin password was not provided, the admin user will not be created. You will have to set up an admin username/password later using user-seed.conf. Creating unit file... Current splunk is running as non root, which cannot operate systemd unit files. Please create it manually by 'sudo splunk enable boot-start' later. Failed to create the unit file. Please do it manually later. Splunk> Now with more code! sudo -H -u splunk /opt/splunkforwarder/bin/splunk status Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" splunkd is running (PID: 3132350). splunk helpers are running (PIDs: 3132354).

# sudo -H -u splunk /opt/splunkforwarder/bin/splunk stop Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" Stopping splunkd... Shutting down. Please wait, as this may take a few minutes. . [ OK ] Stopping splunk helpers... [ OK ] Done. # /opt/splunkforwarder/bin/splunk enable boot-start -user splunk Systemd unit file installed by user at /etc/systemd/system/SplunkForwarder.service. Configured as systemd managed service. systemctl start SplunkForwarder.service Job for SplunkForwarder.service failed because the control process exited with error code. See "systemctl status SplunkForwarder.service" and "journalctl -xe" for details. systemctl status SplunkForwarder.service ● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start' Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2022-06-21 12:58:55 UTC; 27s ago Process: 3141480 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCES> Process: 3141478 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCESS) Process: 3141477 ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=203/EXEC) Process: 3141475 ExecStartPre=/bin/bash -c chown -R splunk:splunk /opt/splunkforwarder (code=exited, status=0/SUCCESS) Main PID: 3141477 (code=exited, status=203/EXEC) Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'. Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'. Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Service RestartSec=100ms expired, scheduling restart. Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Scheduled restart job, restart counter is at 5. Jun 21 12:58:55 <host> systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'. Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Start request repeated too quickly. Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'. Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

_joe — Tue, 28 Jun 2022 18:12:25 GMT

This warning got added to "known issues" for 9.0.0

Date filed Issue number Description

2022-06-23

SPL-226019

Warning appears in the universal forwarder whenever any spl command is run: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder". This warning is expected and will not affect functionality.

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

mustapha_arakji — Thu, 02 Mar 2023 04:01:17 GMT

https://docs.splunk.com/Documentation/Forwarder/9.0.4/Forwarder/KnownIssues

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

alek_cybersec — Mon, 06 Mar 2023 12:30:02 GMT

And what about this:

@_joe wrote:

I am also finding that "./splunk disable boot-start" does not correctly remove the /etc/init.d/splunk script

I've also noticed similar problem.

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Skeer-Jamf — Mon, 10 Jul 2023 21:11:22 GMT

So a known issue and will not cause problems, that's great. Any idea on a cause? Every time I call the splunk binary whether is starting/restarting the app or invoking btool.

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

thahir — Wed, 20 Dec 2023 11:45:29 GMT

@Skeer-Jamf you got any resolution for this issue

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

jfrench — Wed, 07 Feb 2024 03:08:05 GMT

Try running btool when this "non-impacting" error occurs. It doesn't run.

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Merve — Wed, 31 Jul 2024 08:31:46 GMT

This annoying 'non-impacting' known issue is messing up with my ansible variables under facts.d and eventually all my ansible roles, user creations including splunk user, ldap, etc etc end up in a 'impacting issue' and fatal errors situation.

I test it by using Splunk UF version 8.x in my ansible playbooks - everything is working seamlessly and fine.

What is the fix for this IMPACTING known issue?

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Merve — Wed, 31 Jul 2024 08:37:31 GMT

And what is the fix for that?

Because this annoying error is messing up with ansible variables.
I had to use Splunk UF version 8.x - it works fine.
I had other issues on Splunk Enterprise version 9.x - disappointing

topic Re: Linux UF boot-start issues and non-impacting errors on 9.0.0 in Installation

Linux UF boot-start issues and non-impacting errors on 9.0.0

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0

Re: Linux UF boot-start issues and non-impacting errors on 9.0.0