Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

[whrg]

Hello Splunkers,

So Splunk 7.2.2 was just released and it now brings a systemd service.

However, I noticed that now the splunk system user under Linux cannot start/stop Splunk anymore.

Here is what I mean. This is how I installed Splunk 7.2.2:

root$ rpm -i splunk-7.2.2-48f4f251be37-linux-2.6-x86_64.rpm
root$ /opt/splunk/bin/splunk enable boot-start -user splunk
Init script installed at /etc/systemd/system/.
Init script is configured to run at boot.
root$ systemctl start Splunkd

Now when I switch to the Splunk user and try to restart Splunk, it is asking me for root credentials:

root$ su -l splunk
splunk$ /opt/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password:

Is this intended behavior that I now need root credentials to start/stop Splunk when logged in as the splunk user?

You see, my usual workflow is to log in as the splunk user, make some changes to configuration files (/opt/splunk belongs to splunk) and then restart Splunk. I don't want the splunk user to have sudo rights. This used to work in 7.2.1 and before.

[usd0872]

@dimrirahul : Thanks, the -systemd-managed 0 flag works fine, saves me a lot of trouble. Too bad it is not documented in splunk help enable boot-start.

[jkat54]

When in systemd act like the systemd

Dont put half in SysV and half in SystemD. That just wont work well at all.

[gjanders]

Agreed, however there was no real option until 7.2.x

[jkat54]

I have 6.6.7 in sysd, it’s doable but not out of the box.

[gjanders]

I'm assuming you are using systemctl to stop and start splunk? That would work...

[gjanders]

I believe the suggestion here is that
"once you decide to be systemd's
child, then systemd must be the one to
arbitrate your startup and shutdown".

It's slightly more complicated than this, once your running an OS using systemd, than systemd should be the one to shutdown/startup your process!
On newer Oracle releases (redhat based), they use systemd, init.d exists but it's just a process running under systemd.

If Splunk starts as part of OS boot, no problem, it's a systemd process, however if you run splunk restart, or splunk stop/start on the command line, the process no longer appears as a systemd process (note that I'm using init here, I also tested using systemd config files on Splunk 7.0 with the exact same issue).

The challenge here is that once the OS shutdown kicks in, systemd kills user level processes on shutdown, I spent a number of hours with support cases, trying systemd switches et cetera but I never found a way to stop systemd from killing the process (well it wasn't consistent, but >50% of the time splunk was terminated and warnings about corruption appeared).

Note that the above problem only exists if you restart splunk after boot time, if the splunk was started by init.d (under systemd) or by systemd (using a unit file), then there is no issue as Splunk shutdown is run gracefully, not killed.

Anyway, I among others had an enhancement request in to use systemctl splunk stop/start once you start using systemd as that is the only way to keep systemd happy!

[martin_mueller]

As an aside, don't use 7.2.2 because of a nasty scheduler bug - use 7.2.3 instead.

[whrg]

I have updated to 7.2.3. However, the issue is still present.

[yarick]

If an admin actually started Splunk as root at least once, I would suggest to check permissions on directory $SPLUNK_HOME ( with recursion ), prior to attempting to start Splunk as your user of the limited domain ( in your case, splunk ) ...

[gjanders]

Does the post by twinspop in this thread https://answers.splunk.com/answers/59662/is-there-a-systemd-unit-file-for-splunk.html help?

[whrg]

@gjanders Thanks for the reply!
So I just modified polkit, but it's not making a difference yet.
I will report back if I get it working.

[whrg]

Okay so this solution is not working for me because of this line:

action.lookup("unit") == "Splunkd.service"

After some debugging (on my CentOS 7) I found that action.lookup("unit") always returns "undefined" instead of the name of the systemd unit.

I could find some other people having the same issue:
Polkit / Systemd interaction
Centos/Polkit - allowing user to restart specific service

It seems my version of systemd is too old (systemd 219), even though I'm running an up-to-date CentOS 7.

One comment refers to using sudo instead. Perhaps that's worth a try.