Splunk Search

Splunk 7.2.2 - systemd - Root privileges required when starting/stopping Splunk?

whrg
Motivator

Hello Splunkers,

So Splunk 7.2.2 was just released and it now brings a systemd service.

However, I noticed that now the splunk system user under Linux cannot start/stop Splunk anymore.

Here is what I mean. This is how I installed Splunk 7.2.2:

root$ rpm -i splunk-7.2.2-48f4f251be37-linux-2.6-x86_64.rpm
root$ /opt/splunk/bin/splunk enable boot-start -user splunk
Init script installed at /etc/systemd/system/.
Init script is configured to run at boot.
root$ systemctl start Splunkd

Now when I switch to the Splunk user and try to restart Splunk, it is asking me for root credentials:

root$ su -l splunk
splunk$ /opt/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password:

Is this intended behavior that I now need root credentials to start/stop Splunk when logged in as the splunk user?

You see, my usual workflow is to log in as the splunk user, make some changes to configuration files (/opt/splunk belongs to splunk) and then restart Splunk. I don't want the splunk user to have sudo rights. This used to work in 7.2.1 and before.

1 Solution

jkat54
SplunkTrust
SplunkTrust

Just add all needed commands to sudoers separately:

splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl stop Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl start Splunkd.service 
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl status Splunkd.service

After that, the splunk user in the splunk group, should be able to run systemctl commands from ALL terminals without being prompted for a password.

Now you have to change the admin's runbook a bit so they know to use systemctl, but that is all.

See sudo man pages for more details.

View solution in original post

svarkhede
Splunk Employee
Splunk Employee

Splunk solved this problem in v 8.1.1 by implementing polkit along with Systemd. 

./splunk enable boot-start -systemd-managed 1 -create-polkit-rules 1 -user <username>

Depends on Polkit and systemd version available on your system, Splunk will give granular access to the user mentioned in above command. 

Polkit  helps to not hand out  {{sudo}}  access to the user, so its a helpful feature for systemd.

https://docs.splunk.com/Documentation/Splunk/8.2.2/Admin/RunSplunkassystemdservice#:~:text=file%20na...

 

0 Karma

bandit
Motivator

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0

jonxilinx
Path Finder

worked

no more messy systemd or messing with sudoers 🙂

0 Karma

Preet
Splunk Employee
Splunk Employee

Worked.

 

Tags (1)
0 Karma

jkat54
SplunkTrust
SplunkTrust

Just add all needed commands to sudoers separately:

splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl stop Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl start Splunkd.service 
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl status Splunkd.service

After that, the splunk user in the splunk group, should be able to run systemctl commands from ALL terminals without being prompted for a password.

Now you have to change the admin's runbook a bit so they know to use systemctl, but that is all.

See sudo man pages for more details.

marend
Explorer

splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl stop Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl start Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl status Splunkd.service
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl stop Splunkd
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl start Splunkd
splunk ALL=(root) NOPASSWD: /usr/bin/systemctl status Splunkd
splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk restart
splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk stop
splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk start
splunk ALL=(root) NOPASSWD: /opt/splunk/bin/splunk status

 

sudo systemctl daemon-reload

 

This works just fine!

 

0 Karma

whrg
Motivator

It's too bad that the start-up messages (Checking http port; Checking indexes; Waiting for web server to be available...) are not immediately visible as it used to be.
"journalctl -u Splunkd" works similar but requires adding the splunk user to the group systemd-journal.

0 Karma

whrg
Motivator

Thanks. This helped me a lot. I made two modifications to get it working:

splunk ALL=(root) NOPASSWD: /usr/bin/systemctl restart Splunkd.service
...

For one, the service is called Splunkd instead of splunkd.
Secondly, I changed it from (splunk) to (root). This means that splunk can run this command as the root user.

Now it works when I run:

splunk$ sudo /usr/bin/systemctl restart Splunkd.service

jkat54
SplunkTrust
SplunkTrust

Perfect,
I’ll update my post

jkat54
SplunkTrust
SplunkTrust

You broke my karma rut with the upvote! Many thanks!

0 Karma

whrg
Motivator

I upvoted and accepted your answer. However, for some reason, it was reverted back. I will try again later.

jkat54
SplunkTrust
SplunkTrust

We’ll look into it, thanks!

0 Karma

SNaikwade
Path Finder

Even after updating sudoers file, I am still receiving same error message

0 Karma

fabricebourel
Explorer

It still ask for authentication even using sudo in front of the command ... (splunk version 7.2.3).
I can say I am 100% sure my user is correctly set up "sudowise" and he does not need to enter any password on regular linux command such as "sudo updatebd"... I don' t get it (yet).

0 Karma

vgollapudi
Communicator

It doesn't work for me either.

0 Karma

vgollapudi
Communicator

gjanders, I did upvote your answer. It did work for me.

dwaddle
SplunkTrust
SplunkTrust

I have suspicions, but not a lot of hard proof yet. My guess is that when Splunk 7.2.2 (and later) are running under the systemd unit file provided by Splunk, that calls to splunk stop/start/restart wind up being redirected/proxied to the equivalent systemctl stop/start/restart splunk command.

If you look at the docs on the systemd unit file, you'll see that Splunk requires several specific settings in the unit file:

Type=simple
Restart=always
ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd
Delegate=true

I believe the suggestion here is that "once you decide to be systemd's child, then systemd must be the one to arbitrate your startup and shutdown". Otherwise, how can systemd tell the difference between "I did a restart outside of its purview" and "The process crashed and I need to restart it"

Splunk already does something similar on Windows where the splunk start/stop/restart commands make calls to the Windows Service Control Manager (as it has requirements similar to those of systemd about how services get launched)

In my opinion, the two most unfortunate parts of this are that this came to us via a maintenance release (via 7.2.2, not say 7.3.0), and that there seems to be no cmdline arguments to splunk enable boot-start to say "I would really prefer the legacy init system approach thanks". Hopefully I'm wrong on this last one.

In the meantime, I'll try to get this set up and watch it under strace and see if I can get more proof.

dimrirahul
Explorer

Your suspicions are well founded, this is indeed in line with windows and AIX, splunk tries to detect if it is running under systemd and if it does so correctly proxies calls to systemctl. There is a way to say "I would really prefer the legacy init system approach thanks" use flag -systemd-managed 0 when running enable boot-start in addition to whatever other options you specify.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...