topic Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597) in Getting Data In

Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

martin_mueller — Tue, 17 Sep 2013 10:59:48 GMT

Occasionally, our Windows terminal servers kill the UF service during shutdown, leaving in a stale .pid file behind. This results in Splunk not starting up, requiring manual interaction. With a large number of Windows machines that's not an option, I'm looking for a workaround - Splunk Support currently doesn't have a bug-fix schedule for me.

I see two ways: Either clean up the .pid file through a custom script/whatever, or make the UF shut down more quickly to skirt around the killing of the service during Windows shutdown.
How do you handle this issue?

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

MuS — Tue, 17 Sep 2013 11:56:41 GMT

Hi martin_mueller

you forgot the third option, increasing the waittokill registry entry in Windows. I did not test it, but maybe this would be a way to go for you.

In HEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control you will find the string WaitToKillServiceTimeout.

If you double click it and then in the > Edit String window, change the > Value data from the default of 12000 (12 seconds) to whatever. (Click OK to save the change).

hope this helps...

cheers, MuS

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

martin_mueller — Tue, 17 Sep 2013 14:45:50 GMT

Thanks for your input. That should work on its own, I'll have to see how feasible it is around here to change registry settings on thousands of machines.

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

martin_mueller — Thu, 26 Sep 2013 07:59:35 GMT

In the release notes for 5.0.5 I see this entry:

• „Splunk on Windows does not start/restart properly with deployment server, fails with FATAL loader - Timed out waiting for config lock; see splunkd_stderr.log for details. Exiting. (SPL-70075)

This feels similar to my issue, the logged events in case of a start failure are the same. Can anyone confirm this feeling?

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

martin_mueller — Mon, 30 Sep 2013 13:53:31 GMT

I've done some more testing, and 5.0.5 does not fix this issue. Additionally, I've now found precise steps to reproduce, tested under 5.0.1 and 5.0.5:

Start a UF as a Windows Service
Kill the process
Edit the conf-mutator.pid file in /var/run/splunk to change the PID to the PID of an existing process. This simulates that during the UF being down a different process has been assigned its old PID. During system startup the chances for this are considerable.
Attempt to start the UF service. This will fail with logged events like this: FATAL loader - Timed out waiting for config lock

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

martin_mueller — Mon, 30 Sep 2013 13:59:47 GMT

The key problem here is that during startup the forwarder (both Windows and Linux) do only check whether their old PID exists as a process, but do not check whether that process actually is a Splunk process. As a result, the forwarder believes it already is running if a different process happens to have the old Splunk PID.

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

martin_mueller — Tue, 01 Oct 2013 07:15:35 GMT

I have now tested this with 6.0, and the issue exists there as well.

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

letienne — Thu, 31 Oct 2013 12:29:17 GMT

I have the same issue.

I just opened a support case to see if they have any plan to find a viable solution ([141636])

Kind regards,

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

dshakespeare_sp — Wed, 01 Oct 2014 08:08:21 GMT

http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/6.1.4

Resolved : Startup script should handle stale PID files gracefully after server crashes. (SPL-36597)

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

martin_mueller — Wed, 01 Oct 2014 17:18:20 GMT

Awesome, thanks David!

I don't have the means to test right now, does that fix apply to both full and UF installs on both Windows and Unix platforms?

Re: Looking for a workaround for Windows UFs not starting up after an improper shutdown (SPL-36597)

dshakespeare_sp — Thu, 02 Oct 2014 07:15:27 GMT

This should work for both *ix and Windows I believe