Why does the Splunk Upgrader for Linux 1.0.3 assume that the Splunk user is root if there is a System Unit file?

[BlueSocket]

I have been trying to get the Splunk Remote Upgrader for Linux Universal Forwarders 1.0.3 app to work and upgrade our UFs.

The UFs are installed on a supported version of Rocky Linux (9) and we used the .tgx file to install and upgrade the UFs to 9.4.7. The UFs are running as the "splunk" user using the command below to set up and we set the splunk user to be able to read our resources okay.

splunk enable boot-start -systemd-managed 1 -user splunk

Now, when I use the new app on some of my UFs to Splunk 9.4.8/9, etc, the upgrade seems to complete, but Splunk refuses to start up 3 times and then it rolls back the upgrade to 9.4.7.

In the history folder, I see that the logs look for the System Unit file (/etc/systemd/system/SplunkForwarder.service) and if it exists, then the Upgrader assumes that the user running Splunk is root:root and then uses chown -R root:root to the Splunk folder and then the splunk user does not have access to the folder and the start fails. The section in the splunk_util.sh where I found this is:

        if [ -f "$SPLUNK_CUSTOM_UNIT_FILE_PATH" ] || [ -f "$SPLUNK_DEFAULT_UNIT_FILE_PATH" ]; then

            # if any unit file exists, must run splunk as root/sudo

            print_info "Found Splunk unit file. Will start Splunk as ROOT/SUDO."

            splunk_user="root"

It is not true that if the System Unit file (SplunkForwarder.service) exists, then root must be running Splunk - it contains the lines below that tell systemd to start Splunk as the splunk user and the Upgrader should be looking into the file itself to find the real user.

User=splunk

Group=splunk
root is the user who has access to run "systemctl restart SplunkForwarder", but Splunk is not running as root and never has.

If I can stop the process and prevent the Upgrader from rolling back to 9.4.7, then I can run a disable boot-start and then do a enable boot-start and it works, but that is a hack, personally?!

[richgalloway]

I suggest you submit a case to Splunk Support.  The upgrader should not assume the UF runs as root.  Running as root is a Bad Practice.

[BlueSocket]

Agreed @richgalloway . However, I am not in a position to be able to raise a case with Splunk Support. I am hoping that someone else will read this when they have the same problem and raise the case themselves.

[richgalloway]

If you can't submit a case then submit it as an RFE at https://ideas.splunk.com

[BlueSocket]

Thanks for the suggestion, @richgalloway . Looking back on this, I think that this is an environmental issue and I think that I know why the Upgrader is using the root user for the upgrade - so that it can do the upgrade as root and then put it back to the original user, but for some reason, it is not putting it back.

[richgalloway]

Yes, the upgrader has to run as root to do its work, but the end result should be the UF running as the specified non-root user.  If that is not happening then it's a problem that should be reported using one of the methods previously mentioned.

[BlueSocket]

@richgalloway , I have created an idea, as you suggested:
https://ideas.splunk.com/ideas/APPSID-I-1084