Linux UF boot-start issues and non-impacting errors on 9.0.0

[_joe]

Is anyone else running into boot-start/permissions issues with the 9.0.0 UF running on Linux using init.d scripts for bootstart?

Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"

I am also finding that "./splunk disable boot-start" does not correctly remove the /etc/init.d/splunk script and, contrary to documentation, splunk UF 9.0.0 uses systemd as default.
https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/ConfigureSplunktostartatboottime

Also systemd scripts seem to fail getting the permissions needed even when trying to enable-boot as root. A key error I am seeing is "Failed to create the unit file" when running the install. But it seems to be a total fail.

## When upgrading (from 8.2.5)
runuser -l splunk -c "/opt/splunkforwarder/bin/splunk stop"
tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt
chown -R splunk:splunk /opt/splunkforwarder/
runuser -l splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt"
runuser -l splunk -c "/opt/splunkforwarder/bin/splunk status"
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
(NOTE: Seems to be non-impacting)

### When doing a new install
tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt
chown -R splunk:splunk /opt/splunkforwarder
[root]#  sudo -H -u splunk /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.

IMPORTANT: Because an admin password was not provided, the admin user
 will not be created. You will have to set up an admin username/password
 later using user-seed.conf.
Creating unit file...
Current splunk is running as non root, which cannot operate systemd unit files.
Please create it manually by 'sudo splunk enable boot-start' later.
Failed to create the unit file. Please do it manually later.


Splunk> Now with more code!

sudo -H -u splunk /opt/splunkforwarder/bin/splunk status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
splunkd is running (PID: 3132350).
splunk helpers are running (PIDs: 3132354).

# sudo -H -u splunk /opt/splunkforwarder/bin/splunk stop
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
.                                                          [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.
# /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
Systemd unit file installed by user at /etc/systemd/system/SplunkForwarder.service.
Configured as systemd managed service.


systemctl start SplunkForwarder.service
Job for SplunkForwarder.service failed because the control process exited with error code.
See "systemctl status SplunkForwarder.service" and "journalctl -xe" for details.

systemctl status SplunkForwarder.service
● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
   Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2022-06-21 12:58:55 UTC; 27s ago
  Process: 3141480 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCES>
  Process: 3141478 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCESS)
  Process: 3141477 ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=203/EXEC)
  Process: 3141475 ExecStartPre=/bin/bash -c chown -R splunk:splunk /opt/splunkforwarder (code=exited, status=0/SUCCESS)
 Main PID: 3141477 (code=exited, status=203/EXEC)

Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'.
Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Service RestartSec=100ms expired, scheduling restart.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Scheduled restart job, restart counter is at 5.
Jun 21 12:58:55 <host> systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Start request repeated too quickly.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'.
Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.

[thahir]

@Skeer-Jamf  you got any resolution for this issue

[alek_cybersec]

And what about this: 

---

@_joe wrote:

I am also finding that "./splunk disable boot-start" does not correctly remove the /etc/init.d/splunk script

---

I've also noticed similar problem.

[_joe · Accepted solution]

This warning got added to "known issues" for 9.0.0

Date filed Issue number Description

2022-06-23

SPL-226019

Warning appears in the universal forwarder whenever any spl command is run: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder". This warning is expected and will not affect functionality.

[Merve]

And what is the fix for that?

Because this annoying error is messing up with ansible variables.
I had to use Splunk UF version 8.x - it works fine.
I had other issues on Splunk Enterprise version 9.x - disappointing

[jfrench]

Try running btool when this "non-impacting" error occurs.  It doesn't run.

[mustapha_arakji]

https://docs.splunk.com/Documentation/Forwarder/9.0.4/Forwarder/KnownIssues

[Merve]

This annoying 'non-impacting' known issue is messing up with my ansible variables under facts.d and eventually all my ansible roles, user creations including splunk user, ldap, etc etc end up in a 'impacting issue' and fatal errors situation.

I test it by using Splunk UF version 8.x in my ansible playbooks - everything is working seamlessly and fine.

What is the fix for this IMPACTING known issue?

[Skeer-Jamf]

So a known issue and will not cause problems, that's great.  Any idea on a cause? Every time I call the splunk binary whether is starting/restarting the app or invoking btool.