Is anyone else running into boot-start/permissions issues with the 9.0.0 UF running on Linux using init.d scripts for bootstart?
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
I am also finding that "./splunk disable boot-start" does not correctly remove the /etc/init.d/splunk script and, contrary to documentation, splunk UF 9.0.0 uses systemd as default.
https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/ConfigureSplunktostartatboottime
Also systemd scripts seem to fail getting the permissions needed even when trying to enable-boot as root. A key error I am seeing is "Failed to create the unit file" when running the install. But it seems to be a total fail.
## When upgrading (from 8.2.5)
runuser -l splunk -c "/opt/splunkforwarder/bin/splunk stop"
tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt
chown -R splunk:splunk /opt/splunkforwarder/
runuser -l splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt"
runuser -l splunk -c "/opt/splunkforwarder/bin/splunk status"
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
(NOTE: Seems to be non-impacting)
### When doing a new install
tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt
chown -R splunk:splunk /opt/splunkforwarder
[root]# sudo -H -u splunk /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
This appears to be your first time running this version of Splunk.
IMPORTANT: Because an admin password was not provided, the admin user
will not be created. You will have to set up an admin username/password
later using user-seed.conf.
Creating unit file...
Current splunk is running as non root, which cannot operate systemd unit files.
Please create it manually by 'sudo splunk enable boot-start' later.
Failed to create the unit file. Please do it manually later.
Splunk> Now with more code!
sudo -H -u splunk /opt/splunkforwarder/bin/splunk status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
splunkd is running (PID: 3132350).
splunk helpers are running (PIDs: 3132354).
# sudo -H -u splunk /opt/splunkforwarder/bin/splunk stop
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
. [ OK ]
Stopping splunk helpers...
[ OK ]
Done.
# /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
Systemd unit file installed by user at /etc/systemd/system/SplunkForwarder.service.
Configured as systemd managed service.
systemctl start SplunkForwarder.service
Job for SplunkForwarder.service failed because the control process exited with error code.
See "systemctl status SplunkForwarder.service" and "journalctl -xe" for details.
systemctl status SplunkForwarder.service
● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2022-06-21 12:58:55 UTC; 27s ago
Process: 3141480 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCES>
Process: 3141478 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCESS)
Process: 3141477 ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=203/EXEC)
Process: 3141475 ExecStartPre=/bin/bash -c chown -R splunk:splunk /opt/splunkforwarder (code=exited, status=0/SUCCESS)
Main PID: 3141477 (code=exited, status=203/EXEC)
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'.
Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Service RestartSec=100ms expired, scheduling restart.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Scheduled restart job, restart counter is at 5.
Jun 21 12:58:55 <host> systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Start request repeated too quickly.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'.
Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
This warning got added to "known issues" for 9.0.0
Date filed Issue number Description
2022-06-23 | SPL-226019 | Warning appears in the universal forwarder whenever any spl command is run: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder". This warning is expected and will not affect functionality. |
@Skeer-Jamf you got any resolution for this issue
And what about this:
@_joe wrote:
I am also finding that "./splunk disable boot-start" does not correctly remove the /etc/init.d/splunk script
I've also noticed similar problem.
This warning got added to "known issues" for 9.0.0
Date filed Issue number Description
2022-06-23 | SPL-226019 | Warning appears in the universal forwarder whenever any spl command is run: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder". This warning is expected and will not affect functionality. |
And what is the fix for that?
Because this annoying error is messing up with ansible variables.
I had to use Splunk UF version 8.x - it works fine.
I had other issues on Splunk Enterprise version 9.x - disappointing
Try running btool when this "non-impacting" error occurs. It doesn't run.
This annoying 'non-impacting' known issue is messing up with my ansible variables under facts.d and eventually all my ansible roles, user creations including splunk user, ldap, etc etc end up in a 'impacting issue' and fatal errors situation.
I test it by using Splunk UF version 8.x in my ansible playbooks - everything is working seamlessly and fine.
What is the fix for this IMPACTING known issue?
So a known issue and will not cause problems, that's great. Any idea on a cause? Every time I call the splunk binary whether is starting/restarting the app or invoking btool.