Splunkforwarder 7.3.3 linux enable boot-start bug

[janglin]

What is the point of the "splunk enable boot-start" enabling then immediately disabling the service at boot-start?

cat /etc/system-release

Oracle Linux Server release 7.6

/opt/splunkforwarder/bin/splunk version

Splunk Universal Forwarder 7.3.3 (build 7af3758d0d5e)

systemctl enable splunk.service

Created symlink from /etc/systemd/system/multi-user.target.wants/splunk.service to /usr/lib/systemd/system/splunk.service.

reboot

...

/opt/splunkforwarder/bin/splunk status

splunkd is running (PID: 1772).
splunk helpers are running (PIDs: 1776).

/opt/splunkforwarder/bin/splunk enable boot-start

Note: Forwarding request to 'systemctl enable splunk.service'.
Note: Forwarding request to 'systemctl disable splunk.service'.
Removed symlink /etc/systemd/system/multi-user.target.wants/splunk.service.
Init script installed at /etc/init.d/splunk.
Init script is not configured to run at boot.

reboot

...

/opt/splunkforwarder/bin/splunk status

splunkd is not running.

[bandit]

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0

[BainM · Accepted solution]

Hi janglin-

Splunk is in a tough spot right now regarding this. Splunk needs to support both types of system start ups while acknowledging that a good chunk of the industry is slowing moving toward SystemD.
There's some good documentation on it now here (apologies if you already saw this).

Here's another page (older) that enables init.d boot at start time.

I hope this helps.

[janglin]

(sorry about the font ... it cut&pasted that way)

[janglin]

It is a real pain for automation if we have to check whether a client is using systemd or init.d in order to configure the splunkforwarder to start properly on bootup. This should be something that the "splunk enable boot-start" command figures out automagically.

[janglin]

/opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 0
Note: Forwarding request to 'systemctl enable splunk.service'.
Created symlink from /etc/systemd/system/multi-user.target.wants/splunk.service to /usr/lib/systemd/system/splunk.service.
Note: Forwarding request to 'systemctl disable splunk.service'.
Removed symlink /etc/systemd/system/multi-user.target.wants/splunk.service.
Init script installed at /etc/init.d/splunk.
Init script is not configured to run at boot.

/opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 1
splunk is currently running, please stop it before running enable/disable boot-start

/opt/splunkforwarder/bin/splunk stop
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
[ OK ]
Stopping splunk helpers...
[ OK ]
Done.

/opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 1
Initd script /etc/init.d/splunk exists. splunk is currently enabled as init.d bootstart service.
Please run "splunk disable boot-start" first to disable it as init.d boot-start service

/opt/splunkforwarder/bin/splunk disable boot-start
Disabled.

/opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 1
Systemd unit file installed at /etc/systemd/system/SplunkForwarder.service.
Configured as systemd managed service.

/opt/splunkforwarder/bin/splunk status
splunkd is not running.

reboot ...

/opt/splunkforwarder/bin/splunk status
splunkd is running (PID: 1221).
splunk helpers are running (PIDs: 2324).