topic Re: Major boot-start change with 7.2.3 in Splunk Dev

Major boot-start change with 7.2.3

satyenshah — Mon, 04 Feb 2019 17:21:47 GMT

Update: Trick to force bootstarting with init.d instead of systemd: splunk enable boot-start -systemd-managed 0 (credit goes to @dimrirahul below)

Update #2 (2019-07): with version 7.3.0, the default mode for boot-start reverted back from systemd back to init.d

Splunk snuck a major change in a minor release. Updating from 7.2.1 to 7.2.3 changes the way splunk enable boot-startis processed on Nix. Instead of installing scripts in /etc/init.d, enabling boot-start now installs scripts in /etc/systemd. According to documentation, the installation falls back to init.d if the OS does not offer systemd, but there's no documented way for the user to force init.d on an OS that supports systemd.

In some environments, the scripts may prove buggy. If the user-account you are running splunkd under is associated with a nonstandard group, splunk startup will fail with error Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.

There doesn't appear to be a way to specify the groupname at installation. The command splunk enable boot-start -user <username> throws errors with parameter -user <username>:<groupname>. That syntax is not supported. The workaround is to manually edit the file /etc/systemd/system/SplunkForwarder.service and change:

ExecStartPost=/bin/bash -c "chown -R username:username /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R username:username /sys/fs/cgroup/memory/system.slice/%n"

to:

ExecStartPost=/bin/bash -c "chown -R username:groupname /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R username:groupname /sys/fs/cgroup/memory/system.slice/%n"

Which can be scripted by running sed -i 's/username:username/username:groupname/g' /etc/systemd/system/SplunkForwarder.service' followed by systemctl daemon-reload and systemctl restart SplunkForwarder.service assuming it's a UF.

Documentation reference:
https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/RunSplunkassystemdservice

also informative:
https://www.duanewaddle.com/splunk-7-2-2-and-systemd/

Re: Major boot-start change with 7.2.3

woodcock — Mon, 04 Feb 2019 18:29:34 GMT

It is a total disaster. There are currently no valid and secure configurations to make this new systemd method work. To make it worse, as you pointed out, there doesn't seem to be a supported way to make the tried and true init.d method work. Splunk is well aware of the situation. Version 7.2.4 should be dropping any day now I and I fully expect the entire "feature" to be removed. We shall see.

My guidance to clients right now is to avoid all versions of the 7.2.* stream like the plague and wait for 7.2.4, which probably will be production ready.

Re: Major boot-start change with 7.2.3

davemackey — Tue, 05 Feb 2019 18:19:44 GMT

I'm happy that strides are being made to conform to the Systemd architecture. I noticed long ago that the automatic boot functions used the /etc/init.d scripts - so I ended up writing my own for RHEL 7 servers. You can use the following as "/etc/systemd/system/SplunkForwarder.service" without having to use Splunk's boot-start at all:

[Unit]
Description=Splunk Universal Forwarder
After=network-online.target rsyslog.service

[Service]
User=splunk
Group=splunk
ExecStart=/opt/splunkforwarder/bin/splunk start
ExecStop=/opt/splunkforwarder/bin/splunk stop
StandardOutput=syslog
Type=oneshot
RemainAfterExit=yes

[Install]
Alias=splunk.service
WantedBy=multi-user.target
WantedBy=graphical.target

Re: Major boot-start change with 7.2.3

gjanders — Thu, 07 Feb 2019 04:57:17 GMT

This is only partially Splunk's issue, the systemd replacement for init.d is complicated and has it's own flaws.

However if Splunk wanted to introduce systemd support, they should have added a flag to run the systemctl commands with a sudo option, as most distributions do not have any other option! (Redhat 7 for example)

I have an open case on this issue currently as I've had issues with the init.d scripts (due to the OS systemd causing issues), and now I have issues with the new Splunk systemd support not quite working either!

Re: Major boot-start change with 7.2.3

dimrirahul — Sat, 09 Feb 2019 01:13:48 GMT

Actually there is a way to force init.d use flag "-systemd-managed 0" when running enable boot-start, this may be less documented. Essentially if this flag is not present splunk tries to detect the environment that it is running in and tries to do the right thing, if you set this flag to 0, it overrides the detection and tries to install script using init.d flavor if you set it to '1' then it just tries to create a unit file even though your system may not be running systemd. Hope this was useful in some way

Re: Major boot-start change with 7.2.3

dimrirahul — Sat, 09 Feb 2019 01:19:54 GMT

@gjanders ,
The way its envisioned is that setting up of any systemd service is an systemd admin action, more specifically when you configure splunk to run you must have authorization to make that change, and then systemd takes care of managing lifecycle of splunk service. There is also the notion of who the service is being run as, for example "Joe", you could specify this with the -user flag.
Kindly look at my response above how to co-erce splunk to choose what configuration type to use(systemd vs init.d)
Thanks

Re: Major boot-start change with 7.2.3

MuS — Sat, 09 Feb 2019 03:39:53 GMT

Thanks for this, I notified the docs team to add this to the docs.

cheers, MuS

Re: Major boot-start change with 7.2.3

satyenshah — Sat, 09 Feb 2019 06:12:32 GMT

Workaround posted by @dimrirahul above. Command to force init.d: splunk enable boot-start -systemd-managed 0

Re: Major boot-start change with 7.2.3

gjanders — Sat, 09 Feb 2019 07:30:55 GMT

@dimrirahul thanks for the information. My Linux OS using init.d (which is under systemd) has issues with Splunk stop under some circumstances, I was hoping upgrading to 7.2.3 + systemd would help this but I still have some complications here...if Splunk adds sudo into the new systemd feature then I believe it will resolve my issues.

I have a case open with support and will see how that goes.

Re: Major boot-start change with 7.2.3

woodcock — Sat, 09 Feb 2019 16:20:09 GMT

It IS Splunk's problem because they did not offer a way to do it either way. They got rid of a tried-and-true way (yes, with it's own well-understood minor flaws) and replaced it with a giant security hole.

Re: Major boot-start change with 7.2.3

smitra_splunk — Tue, 05 Mar 2019 18:32:00 GMT

its finally official in 7.2.4 docs
https://docs.splunk.com/Documentation/Splunk/7.2.4/Admin/RunSplunkassystemdservice

Additional options for enable boot-start
The enable boot-start command supports these additional options in version 7.2.2 and later:

Install splunk init script
In version 7.2.2 and later, enable boot-start includes a -systemd-managed 0|1 option that controls whether to install the splunk init script in /etc/init.d or the splunkd unit file in /etc/systemd/system. If you do not specify the -systemd-managed option, enable boot-start defaults to -systemd-managed 1 and installs the splunkd unit file.

To override the default and install the splunk init script, specify -systemd-managed 0. For example:

$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 0 -user <username>
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.

Re: Major boot-start change with 7.2.3

mikecarden — Thu, 30 May 2019 17:28:17 GMT

Don't forget that if you have boot-start enabled by one method and you want to swap to the other then you have to run two commands:-

splunk disable boot-start
splunk enable boot-start -systemd-managed 0

or visa-versa to swap to systemctl from initd scripts

Re: Major boot-start change with 7.2.3

gjanders — Thu, 30 May 2019 21:56:38 GMT

Also refer to Splunk systemd unit file in versions 7.2.2 and newer - how do I stop this prompting for the root password? (Q&A)

Re: Major boot-start change with 7.2.3

bandit — Tue, 31 Dec 2019 18:56:55 GMT

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0