Getting Data In

Universal Forwarder Using High CPU?

skoelpin
SplunkTrust
SplunkTrust

I recently installed a Universal Forwarder on an HA Windows server the other day and the guy who owns the server was complaining the CPU has nearly maxed out and shut down the box. He then took a shot in the dark and turned the Splunk service off and the CPU dropped, so it's pretty safe to assume it was the UF causing the problem. I'm a little surprised with this since it's a universal forwarder which has a small footprint for resource utilization.

So my question is, can someone point me in the right direction to find out why this happened? Does Splunk log the CPU utilization? Are there any logs I can look at to see what the issue is? Has anyone else experienced this issue before? Could it be misconfigured?

0 Karma
1 Solution

skoelpin
SplunkTrust
SplunkTrust

It's running 6.4.1

View solution in original post

0 Karma

patrickernsten
Explorer

Problems is fixed with removing definition for WMI from scripts.

Under the “$SPLUNK_HOME\bin\scripts” we saw all proceses from the scripts with a High CPU usages.
We comment those processes out for the script definition (these processes are not used for Citrix in this customer environment)
The only proces that we need in the “$SPLUNK_HOME\bin\scripts” is the “splunk-winevtlog.exe”.
After this change the avg CPU of Splunk process are not above 3-4%. And we still receive all the data.

shawngarrettsgp
Path Finder

Any detail for those on later versions? For example I am having load issues on Windows UF's on version 6.4.5 currently consuming a cpu core.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Got word back from Splunk support. They confirmed a CPU bug in their 6.4.1 Universal Forwarder. They recommended I try UF v6.3.6 and that worked perfectly. I've been monitoring the server CPU for the past day and we are no longer having the high CPU issue anymore

somesoni2
Revered Legend

How many files are you monitoring? Does this started to happen after an upgrade to 6.4.1?

0 Karma

patrickernsten
Explorer

Number of files are around 2

0 Karma

skoelpin
SplunkTrust
SplunkTrust

This forwarder is monitoring 2 files which grow to ~ 1GB in size each day. I upgraded the indexer from 6.2 to 6.4.1 about a week ago and have over 100 forwarders running 6.1 and 6.2 with no issues. I recently installed the 6.4.1 UF on this server the other day, it did not have a forwarder installed before this.

0 Karma

somesoni2
Revered Legend

It might be a stretch but would it be possible for your install version 6.1/6.2 on this box to see if that resolves the high CPU issue? If that works, you can narrow down the problem to confirm if it's due to 6.4.1 version UF OR something else going on the server.

skoelpin
SplunkTrust
SplunkTrust

That's a good suggestion. I will do that on Monday and report back my results

0 Karma

skoelpin
SplunkTrust
SplunkTrust

@somesoni2 .. You we're correct, installing an older forwarder version fixed the issue

0 Karma

skoelpin
SplunkTrust
SplunkTrust

It's running 6.4.1

0 Karma

pradeepkumarg
Influencer

Support asked us to wait for 6.3.6 and it might as well be case with the 6.4 to wait for the next 6.4.x for the fix to get in as 6.3.6 is still due to be released.

skoelpin
SplunkTrust
SplunkTrust

I just opened a support case to see if this is the issue. Thanks for your input

The forwarders are backwards compatible right? My indexer is running 6.4.1, but I should be able to install a 6.2 UF?

0 Karma

pradeepkumarg
Influencer

I believe you should be able to. I would cross check the documentation once though. Just wondering, do you have wild cards in your monitor path? Removing the wildcards might have a positive impact.

somesoni2
Revered Legend

They can but A Splunk best practice is to have indexers be at the same or higher version of Splunk Enterprise than the forwarders they receive data from.
https://docs.splunk.com/Documentation/Forwarder/6.4.1/Forwarder/Compatibilitybetweenforwardersandind...

pradeepkumarg
Influencer

What version of the splunk forwarder? We had similar issue when we upgraded to 6.3.1 and had to roll back the upgrade.

patrickernsten
Explorer

We have release 6.5

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...