Universal forwarder version 10 memory leak

[Ixionz]

I am currently in the testing phase of getting our universal forwarders to a more standardized version (either 9.4.4 or version 10), however when I roll out the new version to any VM's splunkforwarder chews up over 80% of memory which causes overall memory utilization to be around 100%  constantly which I am forced to rollback to version 9.4.4

Nothing has been changed at all except the version. 

Is anyone else experiencing similar behavior when they upgrade to version 10 or even do a new install, or has anyone else seen this behavior out there (not necessarily VM's but maybe physical boxes) as i don't want to roll something out to our environment and causes more problems than solutions.

[darren]

I see this article: https://splunk.my.site.com/customer/s/article/High-CPU-and-Memory-Usage-After-Splunk-UF-10-Upgrade

I've just tested out 9.4.5.0, and having same issue with it crashing servers. 

9.4.4.0 seems to be safe for us.

So far we've seen these crashes on Windows 2016, not sure if it affects other OS versions or not.

[darren]

I was told to try the fix in:

https://community.splunk.com/t5/Splunk-Enterprise/URGENT-All-splunk-forwarders-upgraded-to-10-0-vers...

"Disabled the

evt_resolve_ad_obj = 0 

in Splunk_TA_windows app , logs have now ceased. "

For reference, this is the ticket I made.  Luckily, we were able to catch this issue in dev before deploying 10.0.0.0 to prod.

https://community.splunk.com/t5/Splunk-Enterprise/In-UniversalForwarder-10-0-0-0-splunk-winevtlog-ex...

[darren]

However, if we do the below "fix", then AD SID and AD GUID strings won't be resolved to the actual AD names, which would be really annoying.  I think we're going to hold off on 10.0.0.0 until the evt_resolve_ad_obj feature is fixed and working again without crashing our servers.

[WinEventLog://Security] stanzas inside of inputs.conf:

evt_resolve_ad_obj = 0

[isoutamo]

Please create a support ticket.

Anyhow it's best practice to wait something like X.0.3 or even X.1.2 or similar before go into production. There have been almost every time when a new version has launched more or less nasty and critical bugs.

[livehybrid]

Hi @Ixionz 

Are you able to confirm please the name of the process(es) running which consume this amount of memory? And also the total amount of memory on these VMs?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Dave737]

I upgraded to Splunk Forwarder 10.0.1 yesterday on a PC running Windows10 with 32GB of RAM. The process name is "Monitor windows event log" which is called from "splunk-wineventlog.exe" This process sat consuming over 28GB of RAM! 

I reverted back to 9.4.3 which consumes about 150MB of RAM.

This seems to affect physical servers, VM's and PC's. Luckily I didn't deploy it to too many machines and it's strange that some are running the update with no memory issues as yet. I have had to revert the forwarder on 3 machines but still testing on half a dozen others.