Splunk Search

Systemd support with Splunk does not work on SLES

dchoi_splunk
Splunk Employee
Splunk Employee

When we set up Splunk to start under systemd it prompts us recursively for the root password even we're running Splunk as root Or we're running under sudo.

$SPLUNK_HOME/bin/splunk enable boot-start -user splunk
$SPLUNK_HOME/bin/splunk start

Tags (1)
0 Karma

bandit
Motivator

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0

dchoi_splunk
Splunk Employee
Splunk Employee

After enabling auto-start under systemd : no issue here
$SPLUNK_HOME/bin/splunk enable boot-start -user splunk
$SPLUNK_HOME/bin/splunk start

Starting splunk via systemctl from the root user works as expected
Starting splunk as per the doco ($SPLUNK_HOME/bin/splunk start) as below,
https://docs.splunk.com/Documentation/Splunk/latest/Admin/RunSplunkassystemdservice#Configure_system...

To start splunkd.
[sudo] $SPLUNK_HOME/bin/splunk start
This starts splunkd as a systemd service.

Getting into the following:

Stopped helpers.
Removing stale pid file... done.
Splunk> The Notorious B.I.G. D.A.T.A.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-7.2.3-06d57c595b80-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'Splunkd.service'.
Authenticating as: root
Password:

Once you're running into the issue, you'll be able to get the Splunk started with below workaround:

Under the path for SLES 12, /etc/polkit-1/rules.d, making a rule for Splunk user and org.freedesktop.systemd1.manage-units as below:

cat /etc/polkit-1/rules.d/10-splunk.rules

polkit.addRule(function(action, subject) {
if(action.id == "org.freedesktop.systemd1.manage-units" && subject.user == "splunk") {
return polkit.Result.YES;
}
});

It would allow the Splunk service to start as normal.
In addition, Splunk will be working further under SPL-164816, which systemd configuration on SLES prompts root password when starting for the fix. Stay tuned.

Spranta
Splunk Employee
Splunk Employee

Hi,
do you have an update regarding that issue? We are having the same problems and the workaround didn't work. 😕
Alex

0 Karma

Spranta
Splunk Employee
Splunk Employee

also not working on SuSe Enterprise Server 12 😞

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...