I recently installed a Universal Forwarder on an HA Windows server the other day and the guy who owns the server was complaining the CPU has nearly maxed out and shut down the box. He then took a shot in the dark and turned the Splunk service off and the CPU dropped, so it's pretty safe to assume it was the UF causing the problem. I'm a little surprised with this since it's a universal forwarder which has a small footprint for resource utilization.
So my question is, can someone point me in the right direction to find out why this happened? Does Splunk log the CPU utilization? Are there any logs I can look at to see what the issue is? Has anyone else experienced this issue before? Could it be misconfigured?
Problems is fixed with removing definition for WMI from scripts.
Under the “$SPLUNK_HOME\bin\scripts” we saw all proceses from the scripts with a High CPU usages.
We comment those processes out for the script definition (these processes are not used for Citrix in this customer environment)
The only proces that we need in the “$SPLUNK_HOME\bin\scripts” is the “splunk-winevtlog.exe”.
After this change the avg CPU of Splunk process are not above 3-4%. And we still receive all the data.
Got word back from Splunk support. They confirmed a CPU bug in their 6.4.1 Universal Forwarder. They recommended I try UF v6.3.6 and that worked perfectly. I've been monitoring the server CPU for the past day and we are no longer having the high CPU issue anymore
This forwarder is monitoring 2 files which grow to ~ 1GB in size each day. I upgraded the indexer from 6.2 to 6.4.1 about a week ago and have over 100 forwarders running 6.1 and 6.2 with no issues. I recently installed the 6.4.1 UF on this server the other day, it did not have a forwarder installed before this.
It might be a stretch but would it be possible for your install version 6.1/6.2 on this box to see if that resolves the high CPU issue? If that works, you can narrow down the problem to confirm if it's due to 6.4.1 version UF OR something else going on the server.
Support asked us to wait for 6.3.6 and it might as well be case with the 6.4 to wait for the next 6.4.x for the fix to get in as 6.3.6 is still due to be released.
I just opened a support case to see if this is the issue. Thanks for your input
The forwarders are backwards compatible right? My indexer is running 6.4.1, but I should be able to install a 6.2 UF?
I believe you should be able to. I would cross check the documentation once though. Just wondering, do you have wild cards in your monitor path? Removing the wildcards might have a positive impact.
They can but A Splunk best practice is to have indexers be at the same or higher version of Splunk Enterprise than the forwarders they receive data from.