I'm unable to get Splunk 4.2 or 4.2.1 working. The splunkd service starts but then stops almost immediately.
Starting splunk server daemon (splunkd)...
Splunkd: Starting (pid 2624)
Timed out waiting for splunkd to start.
splunkweb is already running.
This is on a Windows Server 2003 R2, 64-Bit virtual machine. Prior versions of Splunk have run on this machine for about a year.
I get the same result when upgrading from 4.1.7 and when doing a new install of 4.2 or 4.2.1.
The only entries in var/log/splunk/splunkd.log that look suspicious to me are:
ERROR loader - win-service: Error running pre-flight-checks (_pclose returned 1). ERROR loader - win-service: Here is the output from running pre-flight-checks: ERROR loader - ERROR - Error opening "C:\Program Files\Splunk\var\log\splunk\splunkd.log": The process cannot access the file because it is being used by another process. ERROR loader - ERROR - Error opening "C:\Program Files\Splunk\var\log\splunk\splunkd.log": The process cannot access the file because it is being used by another process.
The Error opening line repeats about 75 times. I've confirmed that nothing outside of Splunk is accessing that log file.
Any idea on what the problem is or what I can do next?
Is it possible for you to help debug this with Process Monitor from http://www.sysinternals.com ? We haven't been able to repro the issue in-house.
You'll want to start Process Monitor and add the following two filter (Ctrl+L) entries (I'll separate the fields with pipes for clarity):
Clear (Ctrl+X) the ProcMon log, and then start up Splunk as normal, and after the problem shows up, Save the events to a file (native PML format is fine). I don't think you can upload files to Answers, but you can create a case at the Splunk Support portal and then attach a file to your case.
Thanks. We'll take a look at the PML file - in the meantime, can you also attach any Splunk-*.log files in %TEMP%? There should be at least one left behind by the installer's attempt to start splunk initially, and this may have more stdout logging.
I don't understand how splunkd.exe can't access splunkd.log file, but it's able to successfully log about not being able to access it !!
Are the splunk services (splunkd, splunkweb) running as a domain user or local system user?
I wonder if this is related to permissions rather then another process accessing the file.
Did you try the upgrade from 4.1.7 to 4.2.x and the new install in the same system?
Do you have another VM you try this in, to see if you can reproduce it?
BTW, the 2nd VM that I tested on was copied from the same template that the existing Splunk server was based on. This template is a plain Windows Server 2003 R2, 64-Bit install with some customization and updates applied.
I assume that this issue is unique to my environment, since no one else seems to be affected, and that it has to do with some change between v4.1 and 4.2. I just wish the logs or someone could tell me what it is.
Thanks for your response, Ledio
I did most of my testing last month with 4.2, so I don't recall exactly what I did, but I tried almost every reasonable combination of domain account vs local system account, same VM vs different VM and upgrade vs new install. v4.1.7 would start, but v4.2 would not.
With the new v4.2.1, I first did an upgrade and when that didn't work, I uninstalled Splunk and tried a new install and when that didn't work, I just rolled back to my snapshot.