topic Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk? in Splunk Search

Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

whrg — Fri, 21 Dec 2018 08:07:03 GMT

Hello Splunkers,

So Splunk 7.2.2 was just released and it now brings a systemd service.

However, I noticed that now the splunk system user under Linux cannot start/stop Splunk anymore.

Here is what I mean. This is how I installed Splunk 7.2.2:

root$ rpm -i splunk-7.2.2-48f4f251be37-linux-2.6-x86_64.rpm
root$ /opt/splunk/bin/splunk enable boot-start -user splunk
Init script installed at /etc/systemd/system/.
Init script is configured to run at boot.
root$ systemctl start Splunkd

Now when I switch to the Splunk user and try to restart Splunk, it is asking me for root credentials:

root$ su -l splunk
splunk$ /opt/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password:

Is this intended behavior that I now need root credentials to start/stop Splunk when logged in as the splunk user?

You see, my usual workflow is to log in as the splunk user, make some changes to configuration files (/opt/splunk belongs to splunk) and then restart Splunk. I don't want the splunk user to have sudo rights. This used to work in 7.2.1 and before.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

gjanders — Fri, 21 Dec 2018 21:45:46 GMT

Does the post by twinspop in this thread https://answers.splunk.com/answers/59662/is-there-a-systemd-unit-file-for-splunk.html help?

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

whrg — Sun, 23 Dec 2018 21:18:10 GMT

@gjanders Thanks for the reply!
So I just modified polkit, but it's not making a difference yet.
I will report back if I get it working.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

whrg — Mon, 24 Dec 2018 09:14:07 GMT

Okay so this solution is not working for me because of this line:

action.lookup("unit") == "Splunkd.service"

After some debugging (on my CentOS 7) I found that action.lookup("unit") always returns "undefined" instead of the name of the systemd unit.

I could find some other people having the same issue:
Polkit / Systemd interaction
Centos/Polkit - allowing user to restart specific service

It seems my version of systemd is too old (systemd 219), even though I'm running an up-to-date CentOS 7.

One comment refers to using sudo instead. Perhaps that's worth a try.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

martin_mueller — Wed, 26 Dec 2018 21:24:44 GMT

As an aside, don't use 7.2.2 because of a nasty scheduler bug - use 7.2.3 instead.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

whrg — Fri, 28 Dec 2018 08:25:19 GMT

I have updated to 7.2.3. However, the issue is still present.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

dwaddle — Mon, 31 Dec 2018 17:02:54 GMT

I have suspicions, but not a lot of hard proof yet. My guess is that when Splunk 7.2.2 (and later) are running under the systemd unit file provided by Splunk, that calls to splunk stop/start/restart wind up being redirected/proxied to the equivalent systemctl stop/start/restart splunk command.

If you look at the docs on the systemd unit file, you'll see that Splunk requires several specific settings in the unit file:

Type=simple
Restart=always
ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd
Delegate=true

I believe the suggestion here is that "once you decide to be systemd's child, then systemd must be the one to arbitrate your startup and shutdown". Otherwise, how can systemd tell the difference between "I did a restart outside of its purview" and "The process crashed and I need to restart it"

Splunk already does something similar on Windows where the splunk start/stop/restart commands make calls to the Windows Service Control Manager (as it has requirements similar to those of systemd about how services get launched)

In my opinion, the two most unfortunate parts of this are that this came to us via a maintenance release (via 7.2.2, not say 7.3.0), and that there seems to be no cmdline arguments to splunk enable boot-start to say "I would really prefer the legacy init system approach thanks". Hopefully I'm wrong on this last one.

In the meantime, I'll try to get this set up and watch it under strace and see if I can get more proof.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

gjanders — Mon, 31 Dec 2018 22:03:12 GMT

I believe the suggestion here is that
"once you decide to be systemd's
child, then systemd must be the one to
arbitrate your startup and shutdown".

It's slightly more complicated than this, once your running an OS using systemd, than systemd should be the one to shutdown/startup your process!
On newer Oracle releases (redhat based), they use systemd, init.d exists but it's just a process running under systemd.

If Splunk starts as part of OS boot, no problem, it's a systemd process, however if you run splunk restart, or splunk stop/start on the command line, the process no longer appears as a systemd process (note that I'm using init here, I also tested using systemd config files on Splunk 7.0 with the exact same issue).

The challenge here is that once the OS shutdown kicks in, systemd kills user level processes on shutdown, I spent a number of hours with support cases, trying systemd switches et cetera but I never found a way to stop systemd from killing the process (well it wasn't consistent, but >50% of the time splunk was terminated and warnings about corruption appeared).

Note that the above problem only exists if you restart splunk after boot time, if the splunk was started by init.d (under systemd) or by systemd (using a unit file), then there is no issue as Splunk shutdown is run gracefully, not killed.

Anyway, I among others had an enhancement request in to use systemctl splunk stop/start once you start using systemd as that is the only way to keep systemd happy!

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

jkat54 — Wed, 02 Jan 2019 18:12:43 GMT

Just add all needed commands to sudoers separately:

splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl stop Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl start Splunkd.service 
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl status Splunkd.service

After that, the splunk user in the splunk group, should be able to run systemctl commands from ALL terminals without being prompted for a password.

Now you have to change the admin's runbook a bit so they know to use systemctl, but that is all.

See sudo man pages for more details.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

jkat54 — Wed, 02 Jan 2019 18:15:17 GMT

When in systemd act like the systemd

Dont put half in SysV and half in SystemD. That just wont work well at all.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

gjanders — Wed, 02 Jan 2019 20:58:57 GMT

Agreed, however there was no real option until 7.2.x

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

whrg — Fri, 04 Jan 2019 10:08:20 GMT

Thanks. This helped me a lot. I made two modifications to get it working:

splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service
...

For one, the service is called Splunkd instead of splunkd.
Secondly, I changed it from (splunk) to (root). This means that splunk can run this command as the root user.

Now it works when I run:

splunk$ sudo /usr/bin/systemctl restart Splunkd.service

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

jkat54 — Fri, 04 Jan 2019 13:28:21 GMT

Perfect,
I’ll update my post

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

jkat54 — Fri, 04 Jan 2019 14:06:49 GMT

You broke my karma rut with the upvote! Many thanks!

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

whrg — Fri, 04 Jan 2019 19:45:03 GMT

I upvoted and accepted your answer. However, for some reason, it was reverted back. I will try again later.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

jkat54 — Sat, 05 Jan 2019 00:07:44 GMT

We’ll look into it, thanks!

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

yarick — Thu, 10 Jan 2019 17:25:59 GMT

If an admin actually started Splunk as root at least once, I would suggest to check permissions on directory $SPLUNK_HOME ( with recursion ), prior to attempting to start Splunk as your user of the limited domain ( in your case, splunk ) ...

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

jkat54 — Wed, 06 Feb 2019 00:33:58 GMT

I have 6.6.7 in sysd, it’s doable but not out of the box.

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

gjanders — Wed, 06 Feb 2019 00:50:15 GMT

I'm assuming you are using systemctl to stop and start splunk? That would work...

Re: Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

dimrirahul — Sat, 09 Feb 2019 07:15:03 GMT

Your suspicions are well founded, this is indeed in line with windows and AIX, splunk tries to detect if it is running under systemd and if it does so correctly proxies calls to systemctl. There is a way to say "I would really prefer the legacy init system approach thanks" use flag -systemd-managed 0 when running enable boot-start in addition to whatever other options you specify.