Update: Trick to force bootstarting with init.d instead of systemd:
splunk enable boot-start -systemd-managed 0 (credit goes to @dimrirahul below)
Update #2 (2019-07): with version 7.3.0, the default mode for boot-start reverted back from systemd back to init.d
Splunk snuck a major change in a minor release. Updating from 7.2.1 to 7.2.3 changes the way
splunk enable boot-startis processed on Nix. Instead of installing scripts in /etc/init.d, enabling boot-start now installs scripts in /etc/systemd. According to documentation, the installation falls back to init.d if the OS does not offer systemd, but there's no documented way for the user to force init.d on an OS that supports systemd.
In some environments, the scripts may prove buggy. If the user-account you are running splunkd under is associated with a nonstandard group, splunk startup will fail with error
Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
There doesn't appear to be a way to specify the groupname at installation. The command
splunk enable boot-start -user <username> throws errors with parameter
-user <username>:<groupname>. That syntax is not supported. The workaround is to manually edit the file /etc/systemd/system/SplunkForwarder.service and change:
ExecStartPost=/bin/bash -c "chown -R username:username /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R username:username /sys/fs/cgroup/memory/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R username:groupname /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R username:groupname /sys/fs/cgroup/memory/system.slice/%n"
Which can be scripted by running
sed -i 's/username:username/username:groupname/g' /etc/systemd/system/SplunkForwarder.service' followed by
systemctl daemon-reload and
systemctl restart SplunkForwarder.service assuming it's a UF.
Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start
systemd defaults to prompting for root credentials upon stop/start/restart of Splunk
Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.
Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):
sudo /opt/splunk/bin/splunk disable boot-start sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0
Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):
sudo /opt/splunkforwarder/bin/splunk disable boot-start sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0
@dimrirahul thanks for the information. My Linux OS using init.d (which is under systemd) has issues with Splunk stop under some circumstances, I was hoping upgrading to 7.2.3 + systemd would help this but I still have some complications here...if Splunk adds sudo into the new systemd feature then I believe it will resolve my issues.
I have a case open with support and will see how that goes.
Don't forget that if you have boot-start enabled by one method and you want to swap to the other then you have to run two commands:-
splunk disable boot-start
splunk enable boot-start -systemd-managed 0
or visa-versa to swap to systemctl from initd scripts
Actually there is a way to force init.d use flag "-systemd-managed 0" when running enable boot-start, this may be less documented. Essentially if this flag is not present splunk tries to detect the environment that it is running in and tries to do the right thing, if you set this flag to 0, it overrides the detection and tries to install script using init.d flavor if you set it to '1' then it just tries to create a unit file even though your system may not be running systemd. Hope this was useful in some way
its finally official in 7.2.4 docs
Additional options for enable boot-start The enable boot-start command supports these additional options in version 7.2.2 and later: Install splunk init script In version 7.2.2 and later, enable boot-start includes a -systemd-managed 0|1 option that controls whether to install the splunk init script in /etc/init.d or the splunkd unit file in /etc/systemd/system. If you do not specify the -systemd-managed option, enable boot-start defaults to -systemd-managed 1 and installs the splunkd unit file. To override the default and install the splunk init script, specify -systemd-managed 0. For example: $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 0 -user <username> Init script installed at /etc/init.d/splunk. Init script is configured to run at boot.
I'm happy that strides are being made to conform to the Systemd architecture. I noticed long ago that the automatic boot functions used the /etc/init.d scripts - so I ended up writing my own for RHEL 7 servers. You can use the following as "/etc/systemd/system/SplunkForwarder.service" without having to use Splunk's boot-start at all:
Description=Splunk Universal Forwarder
It is a total disaster. There are currently no valid and secure configurations to make this new
systemd method work. To make it worse, as you pointed out, there doesn't seem to be a supported way to make the tried and true
init.d method work. Splunk is well aware of the situation. Version 7.2.4 should be dropping any day now I and I fully expect the entire "feature" to be removed. We shall see.
My guidance to clients right now is to avoid all versions of the 7.2.* stream like the plague and wait for 7.2.4, which probably will be production ready.
This is only partially Splunk's issue, the systemd replacement for init.d is complicated and has it's own flaws.
However if Splunk wanted to introduce systemd support, they should have added a flag to run the systemctl commands with a sudo option, as most distributions do not have any other option! (Redhat 7 for example)
I have an open case on this issue currently as I've had issues with the init.d scripts (due to the OS systemd causing issues), and now I have issues with the new Splunk systemd support not quite working either!
It IS Splunk's problem because they did not offer a way to do it either way. They got rid of a tried-and-true way (yes, with it's own well-understood minor flaws) and replaced it with a giant security hole.
The way its envisioned is that setting up of any systemd service is an systemd admin action, more specifically when you configure splunk to run you must have authorization to make that change, and then systemd takes care of managing lifecycle of splunk service. There is also the notion of who the service is being run as, for example "Joe", you could specify this with the -user flag.
Kindly look at my response above how to co-erce splunk to choose what configuration type to use(systemd vs init.d)