Installation

Linux UF boot-start issues and non-impacting errors on 9.0.0

_joe
Communicator

Is anyone else running into boot-start/permissions issues with the 9.0.0 UF running on Linux using init.d scripts for bootstart?

Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"


I am also finding that "./splunk disable boot-start" does not correctly remove the /etc/init.d/splunk script and, contrary to documentation, splunk UF 9.0.0 uses systemd as default.
https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/ConfigureSplunktostartatboottime

Also systemd scripts seem to fail getting the permissions needed even when trying to enable-boot as root. A key error I am seeing is "Failed to create the unit file" when running the install. But it seems to be a total fail.

 

 

## When upgrading (from 8.2.5)
runuser -l splunk -c "/opt/splunkforwarder/bin/splunk stop"
tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt
chown -R splunk:splunk /opt/splunkforwarder/
runuser -l splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt"
runuser -l splunk -c "/opt/splunkforwarder/bin/splunk status"
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
(NOTE: Seems to be non-impacting)

 

### When doing a new install
tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt
chown -R splunk:splunk /opt/splunkforwarder
[root]#  sudo -H -u splunk /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"

This appears to be your first time running this version of Splunk.

IMPORTANT: Because an admin password was not provided, the admin user
 will not be created. You will have to set up an admin username/password
 later using user-seed.conf.
Creating unit file...
Current splunk is running as non root, which cannot operate systemd unit files.
Please create it manually by 'sudo splunk enable boot-start' later.
Failed to create the unit file. Please do it manually later.


Splunk> Now with more code!

sudo -H -u splunk /opt/splunkforwarder/bin/splunk status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
splunkd is running (PID: 3132350).
splunk helpers are running (PIDs: 3132354).

 

# sudo -H -u splunk /opt/splunkforwarder/bin/splunk stop
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
.                                                          [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.
# /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
Systemd unit file installed by user at /etc/systemd/system/SplunkForwarder.service.
Configured as systemd managed service.


systemctl start SplunkForwarder.service
Job for SplunkForwarder.service failed because the control process exited with error code.
See "systemctl status SplunkForwarder.service" and "journalctl -xe" for details.

systemctl status SplunkForwarder.service
● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
   Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2022-06-21 12:58:55 UTC; 27s ago
  Process: 3141480 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCES>
  Process: 3141478 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCESS)
  Process: 3141477 ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=203/EXEC)
  Process: 3141475 ExecStartPre=/bin/bash -c chown -R splunk:splunk /opt/splunkforwarder (code=exited, status=0/SUCCESS)
 Main PID: 3141477 (code=exited, status=203/EXEC)

Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'.
Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Service RestartSec=100ms expired, scheduling restart.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Scheduled restart job, restart counter is at 5.
Jun 21 12:58:55 <host> systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Start request repeated too quickly.
Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'.
Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.



Labels (2)
1 Solution

_joe
Communicator

This warning got added to "known issues" for 9.0.0

Date filed Issue number Description

2022-06-23SPL-226019Warning appears in the universal forwarder whenever any spl command is run: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder". This warning is expected and will not affect functionality.

View solution in original post

thahir
Path Finder

@Skeer-Jamf  you got any resolution for this issue

0 Karma

alek_cybersec
Explorer

And what about this: 


@_joe wrote:


I am also finding that "./splunk disable boot-start" does not correctly remove the /etc/init.d/splunk script


I've also noticed similar problem.

0 Karma

_joe
Communicator

This warning got added to "known issues" for 9.0.0

Date filed Issue number Description

2022-06-23SPL-226019Warning appears in the universal forwarder whenever any spl command is run: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder". This warning is expected and will not affect functionality.

jfrench
Loves-to-Learn Lots

Try running btool when this "non-impacting" error occurs.  It doesn't run.

0 Karma

Skeer-Jamf
Path Finder

So a known issue and will not cause problems, that's great.  Any idea on a cause? Every time I call the splunk binary whether is starting/restarting the app or invoking btool.

0 Karma
Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...