topic Re: Version 7.2.6 - how to verify systemctl restart|stop splunkd is shutting down gracefully? in Monitoring Splunk

Version 7.2.6 - how to verify systemctl restart|stop splunkd is shutting down gracefully?

Glasses — Sat, 06 Jun 2020 02:03:53 GMT

The linux admin chose to use the systemctl option for controlling stop|start|restart splunkd as described here...

https://docs.splunk.com/Documentation/Splunk/7.3.0/Admin/RunSplunkassystemdservice#Configure_systemd_using_enable_boot-start

However, now when I do a systemctl restart splunkd, the shutdown and restart is so fast it makes me think it is not gracefully shutting down and finishing whatever the splunk node was processing at the time.

Besides checking journalctl -u splunkd for logs, is there a way to verify we are not losing data or data processing during a stop|restart?

Is there a way to make systemctl stop|restart|start splunkd verbose?

In 7.3.0 docs it appears there is an indication to choose init script or splunkd.service, is there a benefit to one over the other ? is init more compatible?

Re: Version 7.2.6 - how to verify systemctl restart|stop splunkd is shutting down gracefully?

woodcock — Fri, 14 Jun 2019 15:45:04 GMT

You are totally correct. Splunk TOTALLY borked the systemd unit files and indeed it is force killing splunk, which is a VERY dangerous thing to do on indexers. That is why in version 7.3 (just released), they switched back to init.d boot start as the default. This is an ongoing tragedy that is mostly documented in the 3 links below the the bottom line for now is there is NO WAY to make systemd work that is as secure as it should be (e.g. without modifying polkit to be too permissive) so for now, stick to the init.d method by using the -systemd-managed 0 argument.

https://www.duanewaddle.com/splunk-7-2-2-and-systemd/
https://github.com/duckfez/splunk_systemd
https://answers.splunk.com/answers/738877/splunk-systemd-unit-file-in-versions-722-and-newer.html

Re: Version 7.2.6 - how to verify systemctl restart|stop splunkd is shutting down gracefully?

livesplunkcomsk — Fri, 14 Jun 2019 16:53:23 GMT

So adding the below options to the Service stanza in the systemd unit file does not actually allow graceful shutdown? There is definitely a difference with the time it takes to execute systemctl restart splunkd when the below options are set.

KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=10min

Re: Version 7.2.6 - how to verify systemctl restart|stop splunkd is shutting down gracefully?

woodcock — Sat, 15 Jun 2019 22:17:25 GMT

It does but, if you read the links, that is only 1 of a significant list of very serious problems with the systemd implementation for splunk.

Re: Version 7.2.6 - how to verify systemctl restart|stop splunkd is shutting down gracefully?

jazzypai — Mon, 17 Jun 2019 12:25:55 GMT

Hey @woodcock, do you know where the documentation is referencing the change to boot start?

Re: Version 7.2.6 - how to verify systemctl restart|stop splunkd is shutting down gracefully?

woodcock — Mon, 17 Jun 2019 13:46:09 GMT

I could not find it in the documentation but try it and you will see with your own eyes.

Re: Version 7.2.6 - how to verify systemctl restart|stop splunkd is shutting down gracefully?

bandit — Tue, 31 Dec 2019 19:01:05 GMT

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0