SPlunkd CPU locked out at 100% on 2008 machine only

[dbutch1976]

Hello,

I have a script that rolls out Splunk the several machines. The script automatically detects the architecture and installs the correct version of the .msi package (32 or 64 bit). After doing some checking I have discovered that ALL Windows 7 and 2008 machines I have installed to have their CPUs locked at 100% for single processors, 50% for dual processors, 25% for 4 prcoessors etc.

It is the SplunkD service that is locking out the processors. I have installed this on a Windows 7 32 bit machine (with the 32bit version of the msi) and it also had the problem, this leads me to believe it is an issue with 2008+ architecture.

I have installed the exact same version in my home environment (Splunk 4.2.1 build 98164) and I am having no problems with CPUs locking out. This issue is affecting both physical and virtual machines.

I strongly suspect there is some kind of software conflict, such as SEP or altiris client, etc. Can anyone suggest a course of investigation?

[dbutch1976]

I have found the following errors in the splunkd log:

07-27-2011 13:44:36.939 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-regmon.exe" --driver-path "C:\Program Files\Splunk\bin"" splunk-regmon - GetDriverHandle: Unable to install driver.
07-27-2011 13:44:36.939 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-regmon.exe" --driver-path "C:\Program Files\Splunk\bin"" splunk-regmon - run_regmon: Failed to initialize Registry Monitor
07-27-2011 13:44:36.939 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-regmon.exe" --driver-path "C:\Program Files\Splunk\bin"" Using logging configuration at C:\Program Files\Splunk\etc\log-cmdline.cfg.
07-27-2011 13:44:36.939 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-regmon.exe" --driver-path "C:\Program Files\Splunk\bin"" Open SC Manager failed! Error = 5
07-27-2011 13:44:36.939 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-regmon.exe" --driver-path "C:\Program Files\Splunk\bin"" Open SC Manager failed! Error = 5

After restarting the service I still got these errors, and yet the CPU has not spiked yet. In my experience the CPU can take up a few hours before it spikes suddenly. Can someone explain what these errors are and if they are possibly causing my problem?

[dbutch1976]

Thanks for the overwhelming response! 😉

I've done some additional troubleshooting and it appears this is at least partially a permissions issue. Since I have installed via script I have used a program called cacls to grant FC permissions to the splunk install directory and Read permissions to the windows Event logs folder. My script grants these permissions however I have noticed that if I add the Splunk service account to the local administrators group and restart it the problem vanishes.

I view adding a service account to the domain admins group as a huge security hole, and I feel the same way about granting local admin privledges to a service account across my domain, so these are no fixes in my opinion.

What is the Splunkd service touching that requires even more permissions for it to run properly?