topic Re: Splunk Hanging During Restart in Installation

Splunk Hanging During Restart

aelliott — Mon, 28 Sep 2020 15:16:29 GMT

Since installing DB Connect 1.1.1 our Splunk Instance has had issues restarting. It always Hangs and when we restart splunkd and splunkweb, splunk still does not come back and we have to restart the machine itself.

Note: it appears that both services go down and do not come back up.

I saw this in the log at the end of the shutdown process
11-13-2013 14:54:02.757 -0600 WARN ProcessRunner - Process with pid 2352 did not exit within a given grace period after being signaled to exit. Will have to forcibly terminate.
11-13-2013 14:54:02.757 -0600 WARN ProcessRunner - Process with pid 2352 did not forcibly terminate, status 0x0
11-13-2013 14:54:02.757 -0600 INFO IndexProcessor - handleSignal signum=0: Disabling streaming searches.
11-13-2013 14:54:02.757 -0600 INFO IndexProcessor - request state change from=SHUTDOWN_COMPLETE to=SHUTDOWN_SIGNALED
11-13-2013 14:54:02.897 -0600 INFO ShutdownHandler - shutting down level "ShutdownLevel_CallbackRunner"
11-13-2013 14:54:02.897 -0600 INFO ShutdownHandler - Shutdown complete in 37.89 seconds
11-13-2013 14:54:02.897 -0600 INFO loader - All pipelines finished.

Uninstalling DBX gets rid of this issue, but we'd like to still use it.

System Log on the machine states:
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the Splunkd service.

Re: Splunk Hanging During Restart

aelliott — Mon, 18 Nov 2013 22:44:42 GMT

We found the resolution, we Uninstalled DB Connect.

Re: Splunk Hanging During Restart

lukejadamec — Mon, 18 Nov 2013 23:48:12 GMT

After a system restart, dbx works? Were you able to configure inputs with dbx between splunk restarts?

When you restart splunk, are you doing it from the command prompt with administrator permissions? And, if so are there any errors displayed in the command window?

Re: Splunk Hanging During Restart

aelliott — Tue, 19 Nov 2013 14:13:02 GMT

I use the restart feature built within the UI.
DBX worked between restarts yes.
The error logs do not show any errors except the one I posted above, the timeout in the Event Viewer.

Re: Splunk Hanging During Restart

lukejadamec — Tue, 19 Nov 2013 14:16:41 GMT

Restart it from the command prompt and watch for errors:

splunk\bin\splunk restart

Re: Splunk Hanging During Restart

aelliott — Tue, 19 Nov 2013 14:19:39 GMT

We uninstalled it and do not have intentions of installing it again, it has already taken too much of our time, DB Connect that is.
However we will run the restart to be sure that there are no errors on our dev system.

Re: Splunk Hanging During Restart

MartinMcNutt — Tue, 19 Nov 2013 15:39:19 GMT

What I have noticed on a windows box... When you initiate a Splunk restart with DBConnect installed you have a 50/50 chance of it working. DBConnect/Splunk does not properly shutdown the java.exe process which has a lock on the port used by splunk. So when splunk attempts to start again it will not be able to open the required port currently held by the hung java.exe process.

In order to do a proper restart with DBConnect installed you basically have to create a batch job with:

Splunk Stop
<command to kill Java.exe> Eg taskkill /im java.exe /f
Splunk Start

Re: Splunk Hanging During Restart

aelliott — Tue, 19 Nov 2013 15:48:26 GMT

Ouch, that seems like a bug to me.
In fact it looks like they marked this issue as a "Fixed Problem".
On Windows 2008 R2 (64-bit) server, when the splunkd process was killed, the Java Bridge Server continued to run and did not release management port 8089. This prevented Splunk from restarting. (DBX-314)

Re: Splunk Hanging During Restart

araitz — Tue, 19 Nov 2013 17:07:31 GMT

If you have long-running searches in progress (dbmon-tail, for example), the java bridge will not be shut down in an attempt to preserve data integrity. We did fix DBX-314, but there could be other potential issues on Windows that we haven't found yet. It would be great if you could open a support case so we can diagnose what was going on and improve the product.

Re: Splunk Hanging During Restart

MartinMcNutt — Tue, 19 Nov 2013 18:13:35 GMT

Thanks...As soon as we get my Splunk 6.0.0.2 system under control I will revisit and make sure I am running the latest version.

I think splunk support has my name up on the dart board right now. 🙂

Cheers

Re: Splunk Hanging During Restart

aelliott — Tue, 19 Nov 2013 20:10:34 GMT

I have opened a case, we will work with you to figure this out. Thanks!

Re: Splunk Hanging During Restart

aelliott — Tue, 19 Nov 2013 21:48:00 GMT

If the person hitting the restart button through the UI, does not have permissions to the actual server, could there be issues there?

Re: Splunk Hanging During Restart

araitz — Tue, 19 Nov 2013 21:53:22 GMT

I think you mean that the user trying to stop Splunk from the UI is not a Splunk admin. In that case, they shouldn't be able to see the restart button from the UI, but if they somehow can, then no, they should not be able to effect a restart of Splunk.

Re: Splunk Hanging During Restart

aelliott — Tue, 19 Nov 2013 22:19:09 GMT

I am a splunk admin.. anyway.
Here is what is preventing splunk from starting up:
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8090]: already bound
ERROR: The mgmt port [8090] is already bound. Splunk needs to use this port.
Would you like to change ports? [y/n]:

Re: Splunk Hanging During Restart

araitz — Tue, 19 Nov 2013 22:22:43 GMT

Yes, I understand the nature of the problem is that port 8090 (the splunkd port) is still bound to the previous splunk pid. This is something we have seen on Windows in a few different manifestations. We will be happy to work with you to resolve the issue through the support case.

Re: Splunk Hanging During Restart

lukejadamec — Tue, 19 Nov 2013 23:13:19 GMT

I've seen those errors before when I was trying to configure dbx, but they went away once I configured dbx.
When I saw those errors I would answer no, and them I would either restart java from task manager, and or, delete the pid or lock file from splunk\var\run\splunk\

Re: Splunk Hanging During Restart

aelliott — Mon, 09 Dec 2013 16:01:10 GMT

Anti-virus was causing this issue. Our anti-virus was locking indexes and key files that did not allow splunk to operate normally. Once we added exceptions, Splunk worked normally.