topic Re: Getting "Timed out waiting for splunkd to start." from 4.2.x in Monitoring Splunk

Getting "Timed out waiting for splunkd to start." from 4.2.x

arisadmin — Sat, 23 Apr 2011 22:51:11 GMT

I'm unable to get Splunk 4.2 or 4.2.1 working. The splunkd service starts but then stops almost immediately.

Starting splunk server daemon (splunkd)...

Splunkd: Starting (pid 2624)

Timed out waiting for splunkd to start.

splunkweb is already running.

This is on a Windows Server 2003 R2, 64-Bit virtual machine. Prior versions of Splunk have run on this machine for about a year.

I get the same result when upgrading from 4.1.7 and when doing a new install of 4.2 or 4.2.1.

The only entries in var/log/splunk/splunkd.log that look suspicious to me are:

ERROR loader - win-service: Error running pre-flight-checks (_pclose returned 1).  
ERROR loader - win-service: Here is the output from running pre-flight-checks:  
ERROR loader - ERROR - Error opening "C:\Program Files\Splunk\var\log\splunk\splunkd.log": The process cannot access the file because it is being used by another process.  
ERROR loader - ERROR - Error opening "C:\Program Files\Splunk\var\log\splunk\splunkd.log": The process cannot access the file because it is being used by another process.

The Error opening line repeats about 75 times. I've confirmed that nothing outside of Splunk is accessing that log file.

Any idea on what the problem is or what I can do next?

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

Ledio_Ago — Sun, 24 Apr 2011 06:31:47 GMT

Hi "arisadmin"!

I don't understand how splunkd.exe can't access splunkd.log file, but it's able to successfully log about not being able to access it !!

Are the splunk services (splunkd, splunkweb) running as a domain user or local system user?
I wonder if this is related to permissions rather then another process accessing the file.

Did you try the upgrade from 4.1.7 to 4.2.x and the new install in the same system?

Do you have another VM you try this in, to see if you can reproduce it?

Thanks,
Ledio

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

arisadmin — Mon, 25 Apr 2011 01:54:25 GMT

Thanks for your response, Ledio
I did most of my testing last month with 4.2, so I don't recall exactly what I did, but I tried almost every reasonable combination of domain account vs local system account, same VM vs different VM and upgrade vs new install. v4.1.7 would start, but v4.2 would not.

With the new v4.2.1, I first did an upgrade and when that didn't work, I uninstalled Splunk and tried a new install and when that didn't work, I just rolled back to my snapshot.

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

arisadmin — Mon, 25 Apr 2011 01:54:57 GMT

BTW, the 2nd VM that I tested on was copied from the same template that the existing Splunk server was based on. This template is a plain Windows Server 2003 R2, 64-Bit install with some customization and updates applied.

I assume that this issue is unique to my environment, since no one else seems to be affected, and that it has to do with some change between v4.1 and 4.2. I just wish the logs or someone could tell me what it is.

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

amrit — Mon, 25 Apr 2011 22:44:27 GMT

Is it possible for you to help debug this with Process Monitor from http://www.sysinternals.com ? We haven't been able to repro the issue in-house.

You'll want to start Process Monitor and add the following two filter (Ctrl+L) entries (I'll separate the fields with pipes for clarity):

Process Name|contains|splunk|Include

Operation|contains|process|Include

Clear (Ctrl+X) the ProcMon log, and then start up Splunk as normal, and after the problem shows up, Save the events to a file (native PML format is fine). I don't think you can upload files to Answers, but you can create a case at the Splunk Support portal and then attach a file to your case.

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

arisadmin — Tue, 26 Apr 2011 00:37:49 GMT

Yes, I can do that, but it'll take me a day or so.

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

arisadmin — Wed, 27 Apr 2011 23:03:33 GMT

Ok, I've created Case # 58121.

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

amrit — Thu, 28 Apr 2011 17:16:56 GMT

Thanks. We'll take a look at the PML file - in the meantime, can you also attach any Splunk-*.log files in %TEMP%? There should be at least one left behind by the installer's attempt to start splunk initially, and this may have more stdout logging.

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

arisadmin — Thu, 28 Apr 2011 22:29:32 GMT

I had to repeat the install attempt, but the request Splunk-*.log files have been added to the case.

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

arisadmin — Wed, 04 May 2011 16:56:54 GMT

Did you get everything you needed from me?

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

amrit — Thu, 05 May 2011 00:45:15 GMT

Hey sorry, there was an issue a couple of days ago and it appears those files were removed. Can you upload them one more time? 🙂

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

arisadmin — Thu, 19 May 2011 15:56:35 GMT

Did you get the files I re-uploaded on the 11th? Is there anything else you need from me?

Re: Getting "Timed out waiting for splunkd to start." from 4.2.x

sbsbb — Wed, 04 Dec 2013 09:58:09 GMT

I experience the same problem under 5.0.6
Splunkd in complaining not being able to acces splunkd.log, but is writing in it...