Getting Data In

Universal Forwarder Using High CPU?

SplunkTrust
SplunkTrust

I recently installed a Universal Forwarder on an HA Windows server the other day and the guy who owns the server was complaining the CPU has nearly maxed out and shut down the box. He then took a shot in the dark and turned the Splunk service off and the CPU dropped, so it's pretty safe to assume it was the UF causing the problem. I'm a little surprised with this since it's a universal forwarder which has a small footprint for resource utilization.

So my question is, can someone point me in the right direction to find out why this happened? Does Splunk log the CPU utilization? Are there any logs I can look at to see what the issue is? Has anyone else experienced this issue before? Could it be misconfigured?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

It's running 6.4.1

View solution in original post

0 Karma

Explorer

Problems is fixed with removing definition for WMI from scripts.

Under the “$SPLUNK_HOME\bin\scripts” we saw all proceses from the scripts with a High CPU usages.
We comment those processes out for the script definition (these processes are not used for Citrix in this customer environment)
The only proces that we need in the “$SPLUNK_HOME\bin\scripts” is the “splunk-winevtlog.exe”.
After this change the avg CPU of Splunk process are not above 3-4%. And we still receive all the data.

Path Finder

Any detail for those on later versions? For example I am having load issues on Windows UF's on version 6.4.5 currently consuming a cpu core.

0 Karma

SplunkTrust
SplunkTrust

Got word back from Splunk support. They confirmed a CPU bug in their 6.4.1 Universal Forwarder. They recommended I try UF v6.3.6 and that worked perfectly. I've been monitoring the server CPU for the past day and we are no longer having the high CPU issue anymore

SplunkTrust
SplunkTrust

How many files are you monitoring? Does this started to happen after an upgrade to 6.4.1?

0 Karma

Explorer

Number of files are around 2

0 Karma

SplunkTrust
SplunkTrust

This forwarder is monitoring 2 files which grow to ~ 1GB in size each day. I upgraded the indexer from 6.2 to 6.4.1 about a week ago and have over 100 forwarders running 6.1 and 6.2 with no issues. I recently installed the 6.4.1 UF on this server the other day, it did not have a forwarder installed before this.

0 Karma

SplunkTrust
SplunkTrust

It might be a stretch but would it be possible for your install version 6.1/6.2 on this box to see if that resolves the high CPU issue? If that works, you can narrow down the problem to confirm if it's due to 6.4.1 version UF OR something else going on the server.

SplunkTrust
SplunkTrust

That's a good suggestion. I will do that on Monday and report back my results

0 Karma

SplunkTrust
SplunkTrust

@somesoni2 .. You we're correct, installing an older forwarder version fixed the issue

0 Karma

SplunkTrust
SplunkTrust

It's running 6.4.1

View solution in original post

0 Karma

Influencer

Support asked us to wait for 6.3.6 and it might as well be case with the 6.4 to wait for the next 6.4.x for the fix to get in as 6.3.6 is still due to be released.

SplunkTrust
SplunkTrust

I just opened a support case to see if this is the issue. Thanks for your input

The forwarders are backwards compatible right? My indexer is running 6.4.1, but I should be able to install a 6.2 UF?

0 Karma

Influencer

I believe you should be able to. I would cross check the documentation once though. Just wondering, do you have wild cards in your monitor path? Removing the wildcards might have a positive impact.

SplunkTrust
SplunkTrust

They can but A Splunk best practice is to have indexers be at the same or higher version of Splunk Enterprise than the forwarders they receive data from.
https://docs.splunk.com/Documentation/Forwarder/6.4.1/Forwarder/Compatibilitybetweenforwardersandind...

Influencer

What version of the splunk forwarder? We had similar issue when we upgraded to 6.3.1 and had to roll back the upgrade.

Explorer

We have release 6.5

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!