Monitoring Splunk

Getting "Timed out waiting for splunkd to start." from 4.2.x

New Member

I'm unable to get Splunk 4.2 or 4.2.1 working. The splunkd service starts but then stops almost immediately.

Starting splunk server daemon (splunkd)...

Splunkd: Starting (pid 2624)

Timed out waiting for splunkd to start.

splunkweb is already running.

This is on a Windows Server 2003 R2, 64-Bit virtual machine. Prior versions of Splunk have run on this machine for about a year.

I get the same result when upgrading from 4.1.7 and when doing a new install of 4.2 or 4.2.1.

The only entries in var/log/splunk/splunkd.log that look suspicious to me are:

ERROR loader - win-service: Error running pre-flight-checks (_pclose returned 1).  
ERROR loader - win-service: Here is the output from running pre-flight-checks:  
ERROR loader - ERROR - Error opening "C:\Program Files\Splunk\var\log\splunk\splunkd.log": The process cannot access the file because it is being used by another process.  
ERROR loader - ERROR - Error opening "C:\Program Files\Splunk\var\log\splunk\splunkd.log": The process cannot access the file because it is being used by another process.    

The Error opening line repeats about 75 times. I've confirmed that nothing outside of Splunk is accessing that log file.

Any idea on what the problem is or what I can do next?

Tags (4)
0 Karma

Splunk Employee
Splunk Employee

Is it possible for you to help debug this with Process Monitor from http://www.sysinternals.com ? We haven't been able to repro the issue in-house.

You'll want to start Process Monitor and add the following two filter (Ctrl+L) entries (I'll separate the fields with pipes for clarity):

Process Name|contains|splunk|Include

Operation|contains|process|Include

Clear (Ctrl+X) the ProcMon log, and then start up Splunk as normal, and after the problem shows up, Save the events to a file (native PML format is fine). I don't think you can upload files to Answers, but you can create a case at the Splunk Support portal and then attach a file to your case.

New Member

Did you get the files I re-uploaded on the 11th? Is there anything else you need from me?

0 Karma

Splunk Employee
Splunk Employee

Hey sorry, there was an issue a couple of days ago and it appears those files were removed. Can you upload them one more time? 🙂

0 Karma

New Member

Did you get everything you needed from me?

0 Karma

New Member

I had to repeat the install attempt, but the request Splunk-*.log files have been added to the case.

0 Karma

Splunk Employee
Splunk Employee

Thanks. We'll take a look at the PML file - in the meantime, can you also attach any Splunk-*.log files in %TEMP%? There should be at least one left behind by the installer's attempt to start splunk initially, and this may have more stdout logging.

New Member

Ok, I've created Case # 58121.

0 Karma

New Member

Yes, I can do that, but it'll take me a day or so.

0 Karma

Splunk Employee
Splunk Employee

Hi "arisadmin"!

I don't understand how splunkd.exe can't access splunkd.log file, but it's able to successfully log about not being able to access it !!

Are the splunk services (splunkd, splunkweb) running as a domain user or local system user?
I wonder if this is related to permissions rather then another process accessing the file.

Did you try the upgrade from 4.1.7 to 4.2.x and the new install in the same system?

Do you have another VM you try this in, to see if you can reproduce it?

Thanks,
Ledio

Builder

I experience the same problem under 5.0.6
Splunkd in complaining not being able to acces splunkd.log, but is writing in it...

0 Karma

New Member

BTW, the 2nd VM that I tested on was copied from the same template that the existing Splunk server was based on. This template is a plain Windows Server 2003 R2, 64-Bit install with some customization and updates applied.

I assume that this issue is unique to my environment, since no one else seems to be affected, and that it has to do with some change between v4.1 and 4.2. I just wish the logs or someone could tell me what it is.

0 Karma

New Member

Thanks for your response, Ledio
I did most of my testing last month with 4.2, so I don't recall exactly what I did, but I tried almost every reasonable combination of domain account vs local system account, same VM vs different VM and upgrade vs new install. v4.1.7 would start, but v4.2 would not.

With the new v4.2.1, I first did an upgrade and when that didn't work, I uninstalled Splunk and tried a new install and when that didn't work, I just rolled back to my snapshot.

0 Karma