Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

[whrg]

Hello Splunkers,

So Splunk 7.2.2 was just released and it now brings a systemd service.

However, I noticed that now the splunk system user under Linux cannot start/stop Splunk anymore.

Here is what I mean. This is how I installed Splunk 7.2.2:

root$ rpm -i splunk-7.2.2-48f4f251be37-linux-2.6-x86_64.rpm
root$ /opt/splunk/bin/splunk enable boot-start -user splunk
Init script installed at /etc/systemd/system/.
Init script is configured to run at boot.
root$ systemctl start Splunkd

Now when I switch to the Splunk user and try to restart Splunk, it is asking me for root credentials:

root$ su -l splunk
splunk$ /opt/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password:

Is this intended behavior that I now need root credentials to start/stop Splunk when logged in as the splunk user?

You see, my usual workflow is to log in as the splunk user, make some changes to configuration files (/opt/splunk belongs to splunk) and then restart Splunk. I don't want the splunk user to have sudo rights. This used to work in 7.2.1 and before.

[svarkhede]

Splunk solved this problem in v 8.1.1 by implementing polkit along with Systemd. 

./splunk enable boot-start -systemd-managed 1 -create-polkit-rules 1 -user <username>

Depends on Polkit and systemd version available on your system, Splunk will give granular access to the user mentioned in above command. 

Polkit  helps to not hand out  {{sudo}}  access to the user, so its a helpful feature for systemd.

https://docs.splunk.com/Documentation/Splunk/8.2.2/Admin/RunSplunkassystemdservice#:~:text=file%20na...

[bandit]

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0

[jonxilinx]

worked

no more messy systemd or messing with sudoers 🙂

[Preet]

Worked.

[gjanders]

Please refer to Splunk systemd unit file in versions 7.2.2 and newer - how do I stop this prompting for the root pas... , feel free to up-vote if it helps you.

Thanks

[jkat54 · Accepted solution]

Just add all needed commands to sudoers separately:

splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl stop Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl start Splunkd.service 
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl status Splunkd.service

After that, the splunk user in the splunk group, should be able to run systemctl commands from ALL terminals without being prompted for a password.

Now you have to change the admin's runbook a bit so they know to use systemctl, but that is all.

See sudo man pages for more details.

[marend]

splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl stop Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl start Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl status Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl stop Splunkd
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl start Splunkd
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl status Splunkd
splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk restart
splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk stop
splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk start
splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk status

sudo systemctl daemon-reload

This works just fine!

[whrg]

It's too bad that the start-up messages (Checking http port; Checking indexes; Waiting for web server to be available...) are not immediately visible as it used to be.
"journalctl -u Splunkd" works similar but requires adding the splunk user to the group systemd-journal.

[whrg]

Thanks. This helped me a lot. I made two modifications to get it working:

splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service
...

For one, the service is called Splunkd instead of splunkd.
Secondly, I changed it from (splunk) to (root). This means that splunk can run this command as the root user.

Now it works when I run:

splunk$ sudo /usr/bin/systemctl restart Splunkd.service

[jkat54]

Perfect,
I’ll update my post

[jkat54]

You broke my karma rut with the upvote! Many thanks!

[whrg]

I upvoted and accepted your answer. However, for some reason, it was reverted back. I will try again later.

[jkat54]

We’ll look into it, thanks!

[SNaikwade]

Even after updating sudoers file, I am still receiving same error message

[fabricebourel]

It still ask for authentication even using sudo in front of the command ... (splunk version 7.2.3).
I can say I am 100% sure my user is correctly set up "sudowise" and he does not need to enter any password on regular linux command such as "sudo updatebd"... I don' t get it (yet).

[vgollapudi]

It doesn't work for me either.

[gjanders]

Please refer to Splunk systemd unit file in versions 7.2.2 and newer - how do I stop this prompting for the root pas... , please up-vote if it helps!

[vgollapudi]

gjanders, I did upvote your answer. It did work for me.

[dwaddle]

I have suspicions, but not a lot of hard proof yet. My guess is that when Splunk 7.2.2 (and later) are running under the systemd unit file provided by Splunk, that calls to splunk stop/start/restart wind up being redirected/proxied to the equivalent systemctl stop/start/restart splunk command.

If you look at the docs on the systemd unit file, you'll see that Splunk requires several specific settings in the unit file:

Type=simple
Restart=always
ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd
Delegate=true

I believe the suggestion here is that "once you decide to be systemd's child, then systemd must be the one to arbitrate your startup and shutdown". Otherwise, how can systemd tell the difference between "I did a restart outside of its purview" and "The process crashed and I need to restart it"

Splunk already does something similar on Windows where the splunk start/stop/restart commands make calls to the Windows Service Control Manager (as it has requirements similar to those of systemd about how services get launched)

In my opinion, the two most unfortunate parts of this are that this came to us via a maintenance release (via 7.2.2, not say 7.3.0), and that there seems to be no cmdline arguments to splunk enable boot-start to say "I would really prefer the legacy init system approach thanks". Hopefully I'm wrong on this last one.

In the meantime, I'll try to get this set up and watch it under strace and see if I can get more proof.

[dimrirahul]

Your suspicions are well founded, this is indeed in line with windows and AIX, splunk tries to detect if it is running under systemd and if it does so correctly proxies calls to systemctl. There is a way to say "I would really prefer the legacy init system approach thanks" use flag -systemd-managed 0 when running enable boot-start in addition to whatever other options you specify.