Installation

Splunk Hanging During Restart

aelliott
Motivator

Since installing DB Connect 1.1.1 our Splunk Instance has had issues restarting. It always Hangs and when we restart splunkd and splunkweb, splunk still does not come back and we have to restart the machine itself.

Note: it appears that both services go down and do not come back up.

I saw this in the log at the end of the shutdown process
11-13-2013 14:54:02.757 -0600 WARN ProcessRunner - Process with pid 2352 did not exit within a given grace period after being signaled to exit. Will have to forcibly terminate.
11-13-2013 14:54:02.757 -0600 WARN ProcessRunner - Process with pid 2352 did not forcibly terminate, status 0x0
11-13-2013 14:54:02.757 -0600 INFO IndexProcessor - handleSignal signum=0: Disabling streaming searches.
11-13-2013 14:54:02.757 -0600 INFO IndexProcessor - request state change from=SHUTDOWN_COMPLETE to=SHUTDOWN_SIGNALED
11-13-2013 14:54:02.897 -0600 INFO ShutdownHandler - shutting down level "ShutdownLevel_CallbackRunner"
11-13-2013 14:54:02.897 -0600 INFO ShutdownHandler - Shutdown complete in 37.89 seconds
11-13-2013 14:54:02.897 -0600 INFO loader - All pipelines finished.

Uninstalling DBX gets rid of this issue, but we'd like to still use it.

System Log on the machine states:
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the Splunkd service.

0 Karma
1 Solution

aelliott
Motivator

Anti-virus was causing this issue. Our anti-virus was locking indexes and key files that did not allow splunk to operate normally. Once we added exceptions, Splunk worked normally.

View solution in original post

0 Karma

aelliott
Motivator

Anti-virus was causing this issue. Our anti-virus was locking indexes and key files that did not allow splunk to operate normally. Once we added exceptions, Splunk worked normally.

0 Karma

araitz
Splunk Employee
Splunk Employee

If you have long-running searches in progress (dbmon-tail, for example), the java bridge will not be shut down in an attempt to preserve data integrity. We did fix DBX-314, but there could be other potential issues on Windows that we haven't found yet. It would be great if you could open a support case so we can diagnose what was going on and improve the product.

lukejadamec
Super Champion

I've seen those errors before when I was trying to configure dbx, but they went away once I configured dbx.
When I saw those errors I would answer no, and them I would either restart java from task manager, and or, delete the pid or lock file from splunk\var\run\splunk\

0 Karma

araitz
Splunk Employee
Splunk Employee

Yes, I understand the nature of the problem is that port 8090 (the splunkd port) is still bound to the previous splunk pid. This is something we have seen on Windows in a few different manifestations. We will be happy to work with you to resolve the issue through the support case.

0 Karma

aelliott
Motivator

I am a splunk admin.. anyway.
Here is what is preventing splunk from starting up:
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8090]: already bound
ERROR: The mgmt port [8090] is already bound. Splunk needs to use this port.
Would you like to change ports? [y/n]:

0 Karma

araitz
Splunk Employee
Splunk Employee

I think you mean that the user trying to stop Splunk from the UI is not a Splunk admin. In that case, they shouldn't be able to see the restart button from the UI, but if they somehow can, then no, they should not be able to effect a restart of Splunk.

0 Karma

aelliott
Motivator

If the person hitting the restart button through the UI, does not have permissions to the actual server, could there be issues there?

0 Karma

aelliott
Motivator

I have opened a case, we will work with you to figure this out. Thanks!

0 Karma

MartinMcNutt
Communicator

What I have noticed on a windows box... When you initiate a Splunk restart with DBConnect installed you have a 50/50 chance of it working. DBConnect/Splunk does not properly shutdown the java.exe process which has a lock on the port used by splunk. So when splunk attempts to start again it will not be able to open the required port currently held by the hung java.exe process.

In order to do a proper restart with DBConnect installed you basically have to create a batch job with:

Splunk Stop
<command to kill Java.exe> Eg taskkill /im java.exe /f
Splunk Start

MartinMcNutt
Communicator

Thanks...As soon as we get my Splunk 6.0.0.2 system under control I will revisit and make sure I am running the latest version.

I think splunk support has my name up on the dart board right now. 🙂

Cheers

0 Karma

aelliott
Motivator

Ouch, that seems like a bug to me.
In fact it looks like they marked this issue as a "Fixed Problem".
On Windows 2008 R2 (64-bit) server, when the splunkd process was killed, the Java Bridge Server continued to run and did not release management port 8089. This prevented Splunk from restarting. (DBX-314)

0 Karma

aelliott
Motivator

We uninstalled it and do not have intentions of installing it again, it has already taken too much of our time, DB Connect that is.
However we will run the restart to be sure that there are no errors on our dev system.

0 Karma

lukejadamec
Super Champion

Restart it from the command prompt and watch for errors:

splunk\bin\splunk restart

0 Karma

aelliott
Motivator

I use the restart feature built within the UI.
DBX worked between restarts yes.
The error logs do not show any errors except the one I posted above, the timeout in the Event Viewer.

0 Karma

lukejadamec
Super Champion

After a system restart, dbx works? Were you able to configure inputs with dbx between splunk restarts?

When you restart splunk, are you doing it from the command prompt with administrator permissions? And, if so are there any errors displayed in the command window?

0 Karma

aelliott
Motivator

We found the resolution, we Uninstalled DB Connect.

Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...