Splunk 7.0.1 Init script is not configured to run at boot

[rfitch]

Just upgraded a heavy forwarder to 7.0.1 and I'm not able to enable boot start.

I've tried:
/opt/splunk/bin/splunk enable boot-start
/opt/splunk/bin/splunk enable boot-start -user root

neither seem to work.

[root@dvplagsplnk06 opt]# /opt/splunk/bin/splunk enable boot-start
Note: Forwarding request to 'systemctl enable splunk.service'.
Created symlink from /etc/systemd/system/multi-user.target.wants/splunk.service to /usr/lib/systemd/system/splunk.service.
Note: Forwarding request to 'systemctl disable splunk.service'.
Removed symlink /etc/systemd/system/multi-user.target.wants/splunk.service.
Init script installed at /etc/init.d/splunk.
Init script is not configured to run at boot.

I confirmed with display boot-start and even rebooted the server and the Splunk service doesn't start.

I also tried manually running

systemctl enable splunk.service

with no luck.

I'm on Rhel 7.3
uname -a
Linux dvplagsplnk06 3.10.0-514.26.2.el7.x86_64 #1 SMP Fri Jun 30 05:26:04 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)

[bandit]

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0

[rfitch · Accepted solution]

Follow up from my very excellent call with support

Looks like Chef was laying down a unit file for Splunk that not only conflicted with the Splunk RPM version, but was also incomplete. The fix was to take the unit file from https://answers.splunk.com/answers/590209/rhelcentos-7-systemd-not-honoring-ulimits.html and in my case use it to replace

 /usr/lib/systemd/system/splunk.service 

then run
systemctl daemon-reload
systemctl enable splunk.service

[sloshburch]

Wow. I would have never guessed! Thanks for sharing the solution AND accepting it. This will certainly be informative for others!

[sloshburch]

Sanity check question: Would you disable boot-start and then try to enable again?

[rfitch]

No dice:

[root@nvplagsplnk08 ~]# /opt/splunk/bin/splunk display boot-start 
Init script is not installed (checked: /etc/init.d/splunk).
Init script is not configured to run at boot.
[root@nvplagsplnk08 ~]# /opt/splunk/bin/splunk enable boot-start 
Note: Forwarding request to 'systemctl enable splunk.service'.
Created symlink from /etc/systemd/system/multi-user.target.wants/splunk.service to /usr/lib/systemd/system/splunk.service.
Note: Forwarding request to 'systemctl disable splunk.service'.
Removed symlink /etc/systemd/system/multi-user.target.wants/splunk.service.
Init script installed at /etc/init.d/splunk.
Init script is not configured to run at boot.

[sloshburch]

I notice you ran display, not disable. Good call on opening a case with support in parallel because as you can surmise, you might not be hitting something the rest of the community has run into.

[marycordova]

(ps (pre-script) hope this helps, apologies if I made any assumptions about your technical acumen!)

I don't believe in letting vendors off the hook...but set a boot cron job for now so you're not screwed until this gets fixed:

@reboot /etc/init.d/splunk start (or some such...you may have to prepend that with /bin/bash then the /etc...)

Of course you shouldn't run Splunk as root...but since Splunk elects not to provide a clear mechanism by which to do this, configure your system, still actually monitor logs, and bind to low numbered ports, I'll spare you the lecture.

If you get to the point where you're not running as root you can play around with the options in both the Splunk command to configure boot-start as a specific user or modify the cronjob to run init script as a specific user, example something I have logging to Splunk: @reboot sudo -u username '/opt/something/bin/someotherthing'

Also, I don't know if this was ever fixed but there was a bug where Splunk would ignore the ulimits on a system, so I added them to my init script directly *in both the start and restart functions*:

splunk_start() {
echo Starting Splunk...
ulimit -Hn 65536
ulimit -Sn 32768
"/opt/splunkforwarder/bin/splunk" start --no-prompt --answer-yes

...

splunk_restart() {
echo Restarting Splunk...
ulimit -Hn 65536
ulimit -Sn 32768
"/opt/splunkforwarder/bin/splunk" restart

[sloshburch]

FYI: The specifics on the ulimits are addressed in the docs: http://docs.splunk.com/Documentation/Splunk/latest/Admin/ConfigureSplunktostartatboottime#Enable_boo...