Archive
Highlighted

Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

Motivator

Hello Splunkers,

So Splunk 7.2.2 was just released and it now brings a systemd service.

However, I noticed that now the splunk system user under Linux cannot start/stop Splunk anymore.

Here is what I mean. This is how I installed Splunk 7.2.2:

root$ rpm -i splunk-7.2.2-48f4f251be37-linux-2.6-x86_64.rpm
root$ /opt/splunk/bin/splunk enable boot-start -user splunk
Init script installed at /etc/systemd/system/.
Init script is configured to run at boot.
root$ systemctl start Splunkd

Now when I switch to the Splunk user and try to restart Splunk, it is asking me for root credentials:

root$ su -l splunk
splunk$ /opt/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password:

Is this intended behavior that I now need root credentials to start/stop Splunk when logged in as the splunk user?

You see, my usual workflow is to log in as the splunk user, make some changes to configuration files (/opt/splunk belongs to splunk) and then restart Splunk. I don't want the splunk user to have sudo rights. This used to work in 7.2.1 and before.

Highlighted

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

SplunkTrust
SplunkTrust
Highlighted

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

Motivator

@gjanders Thanks for the reply!
So I just modified polkit, but it's not making a difference yet.
I will report back if I get it working.

Highlighted

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

Motivator

Okay so this solution is not working for me because of this line:

action.lookup("unit") == "Splunkd.service"

After some debugging (on my CentOS 7) I found that action.lookup("unit") always returns "undefined" instead of the name of the systemd unit.

I could find some other people having the same issue:
Polkit / Systemd interaction
Centos/Polkit - allowing user to restart specific service

It seems my version of systemd is too old (systemd 219), even though I'm running an up-to-date CentOS 7.

One comment refers to using sudo instead. Perhaps that's worth a try.

0 Karma
Highlighted

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

SplunkTrust
SplunkTrust

As an aside, don't use 7.2.2 because of a nasty scheduler bug - use 7.2.3 instead.

Highlighted

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

Motivator

I have updated to 7.2.3. However, the issue is still present.

Highlighted

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

Path Finder

If an admin actually started Splunk as root at least once, I would suggest to check permissions on directory $SPLUNK_HOME ( with recursion ), prior to attempting to start Splunk as your user of the limited domain ( in your case, splunk ) ...

0 Karma
Highlighted

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

SplunkTrust
SplunkTrust

I have suspicions, but not a lot of hard proof yet. My guess is that when Splunk 7.2.2 (and later) are running under the systemd unit file provided by Splunk, that calls to splunk stop/start/restart wind up being redirected/proxied to the equivalent systemctl stop/start/restart splunk command.

If you look at the docs on the systemd unit file, you'll see that Splunk requires several specific settings in the unit file:

Type=simple
Restart=always
ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd
Delegate=true

I believe the suggestion here is that "once you decide to be systemd's child, then systemd must be the one to arbitrate your startup and shutdown". Otherwise, how can systemd tell the difference between "I did a restart outside of its purview" and "The process crashed and I need to restart it"

Splunk already does something similar on Windows where the splunk start/stop/restart commands make calls to the Windows Service Control Manager (as it has requirements similar to those of systemd about how services get launched)

In my opinion, the two most unfortunate parts of this are that this came to us via a maintenance release (via 7.2.2, not say 7.3.0), and that there seems to be no cmdline arguments to splunk enable boot-start to say "I would really prefer the legacy init system approach thanks". Hopefully I'm wrong on this last one.

In the meantime, I'll try to get this set up and watch it under strace and see if I can get more proof.

Highlighted

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

SplunkTrust
SplunkTrust

I believe the suggestion here is that
"once you decide to be systemd's
child, then systemd must be the one to
arbitrate your startup and shutdown".

It's slightly more complicated than this, once your running an OS using systemd, than systemd should be the one to shutdown/startup your process!
On newer Oracle releases (redhat based), they use systemd, init.d exists but it's just a process running under systemd.

If Splunk starts as part of OS boot, no problem, it's a systemd process, however if you run splunk restart, or splunk stop/start on the command line, the process no longer appears as a systemd process (note that I'm using init here, I also tested using systemd config files on Splunk 7.0 with the exact same issue).

The challenge here is that once the OS shutdown kicks in, systemd kills user level processes on shutdown, I spent a number of hours with support cases, trying systemd switches et cetera but I never found a way to stop systemd from killing the process (well it wasn't consistent, but >50% of the time splunk was terminated and warnings about corruption appeared).

Note that the above problem only exists if you restart splunk after boot time, if the splunk was started by init.d (under systemd) or by systemd (using a unit file), then there is no issue as Splunk shutdown is run gracefully, not killed.

Anyway, I among others had an enhancement request in to use systemctl splunk stop/start once you start using systemd as that is the only way to keep systemd happy!

0 Karma
Highlighted

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

SplunkTrust
SplunkTrust

When in systemd act like the systemd

Dont put half in SysV and half in SystemD. That just wont work well at all.