topic Splunk service didn't start automatically after server reboot in Splunk Enterprise

Splunk service didn't start automatically after server reboot

mlevsh — Wed, 30 Sep 2020 01:24:10 GMT

Hi,

We have Splunk Enteprise v. 6.6.3 Heavy Forwarders.
1) We configured Splunk to start on reboot as our splunk_user by running

"$SPLUNK_HOME$/bin/splunk enable boot-start -user splunk_user"
2) /etc/init.d/splunk script was created and
SPLUNK_OS_USER was set to splunk_user in /opt/splunk/etc/splunk-launch.conf
3) our heavy forwarders are set to be rebooted once per month and it was going ok .
Splunk services were getting restarted automatically every time

4) Recently we worked with our Linux Admins to increase ulimit -n on one of heavy forwarders.
Our Unix admins made the required change on system level.
They also updated /etc/init.d/splunk script by adding the following to splunk_start() and
splunk_restart() functions:

  ulimit -Hn <value>
  ulimit -Sn  <value>

Then Splunk was stopped and server was rebooted and this time Splunk service didn't start automatically
We had to start it manually.

What could be the reason and what's the best way to troubleshoot this and fix it if needed to prevent this from happening again

Thank you in advance!

Re: Splunk service didn't start automatically after server reboot

ddrillic — Wed, 17 Jul 2019 20:04:26 GMT

We spoke about it at this thread and it might help you - How do we enable a forwarder boot-start?

Re: Splunk service didn't start automatically after server reboot

mlevsh — Wed, 17 Jul 2019 20:21:15 GMT

@ddrillic , thank you!
I actually went through that thread and especially noted your post:
"Very interesting - yesterday we checked a server that had this /etc/init.d/splunk file but the splunk process was not started 15 days ago when the server was rebooted. We ended up rebooting the server a couple of times and in these cases, Splunk did come up. Confusing ; -) "

So it is still not clear for us why splunk auto restart worked before and didn't work this time 🙂

Re: Splunk service didn't start automatically after server reboot

MuS — Wed, 17 Jul 2019 20:27:18 GMT

So, what does your /var/log/messages tell you? Looks like the recent change of /etc/init.d/splunk did not work.

cheers, MuS

Re: Splunk service didn't start automatically after server reboot

ddrillic — Wed, 17 Jul 2019 20:39:04 GMT

Thank you for the attention @mlevsh and the quote ; -)