topic Re: Systemd support with Splunk does not work on SLES in Splunk Search

Systemd support with Splunk does not work on SLES

dchoi_splunk — Tue, 29 Sep 2020 22:49:31 GMT

When we set up Splunk to start under systemd it prompts us recursively for the root password even we're running Splunk as root Or we're running under sudo.

$SPLUNK_HOME/bin/splunk enable boot-start -user splunk
$SPLUNK_HOME/bin/splunk start

Re: Systemd support with Splunk does not work on SLES

dchoi_splunk — Tue, 29 Sep 2020 22:49:33 GMT

After enabling auto-start under systemd : no issue here
$SPLUNK_HOME/bin/splunk enable boot-start -user splunk
$SPLUNK_HOME/bin/splunk start

Starting splunk via systemctl from the root user works as expected
Starting splunk as per the doco ($SPLUNK_HOME/bin/splunk start) as below,
https://docs.splunk.com/Documentation/Splunk/latest/Admin/RunSplunkassystemdservice#Configure_systemd_using_enable_boot-start

To start splunkd.
[sudo] $SPLUNK_HOME/bin/splunk start
This starts splunkd as a systemd service.

Getting into the following:

Stopped helpers.
Removing stale pid file... done.
Splunk> The Notorious B.I.G. D.A.T.A.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-7.2.3-06d57c595b80-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'Splunkd.service'.
Authenticating as: root
Password:

Once you're running into the issue, you'll be able to get the Splunk started with below workaround:

Under the path for SLES 12, /etc/polkit-1/rules.d, making a rule for Splunk user and org.freedesktop.systemd1.manage-units as below:

cat /etc/polkit-1/rules.d/10-splunk.rules

polkit.addRule(function(action, subject) {
if(action.id == "org.freedesktop.systemd1.manage-units" && subject.user == "splunk") {
return polkit.Result.YES;
}
});

It would allow the Splunk service to start as normal.
In addition, Splunk will be working further under SPL-164816, which systemd configuration on SLES prompts root password when starting for the fix. Stay tuned.

Re: Systemd support with Splunk does not work on SLES

Spranta — Tue, 12 Nov 2019 11:04:38 GMT

Hi,
do you have an update regarding that issue? We are having the same problems and the workaround didn't work. 😕
Alex

Re: Systemd support with Splunk does not work on SLES

gjanders — Wed, 13 Nov 2019 01:51:23 GMT

Try Splunk systemd unit file in versions 7.2.2 and newer - how do I stop this prompting for the root password? (Q&A)

Re: Systemd support with Splunk does not work on SLES

Spranta — Wed, 13 Nov 2019 07:48:47 GMT

also not working on SuSe Enterprise Server 12 😞

Re: Systemd support with Splunk does not work on SLES

bandit — Tue, 31 Dec 2019 18:44:48 GMT

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0