UF 10.0 — splunk-winevtlog.exe crashes in VCRUNTIME140.dll / KERNELBASE.dll (EventCode=1000)

[krynol]

After upgrading to UF 10.0 we see many Application Error (EventCode=1000) crashes on a subset of servers only. Faulting modules vary between KERNELBASE.dll (system) and VCRUNTIME140.dll (sometimes loaded from UF bin).

Examples

Faulting app: splunk-winevtlog.exe 2560.0.26759.23473 Faulting module: KERNELBASE.dll 10.0.17763.7553 Exception code: 0xeeab5254 Path: C:\Windows\System32\KERNELBASE.dll

Faulting app: splunk-winevtlog.exe 2560.0.26759.23473 Faulting module: VCRUNTIME140.dll 14.42.34438.0 Path: C:\Program Files\SplunkUniversalForwarder\bin\VCRUNTIME140.dll Exception code: 0xc0000005

Questions

- Does UF 10.0 ship and prefer its own VCRUNTIME140.dll, or should it rely on system VC++ Redistributable?
- Any known compatibility issues with specific KERNELBASE.dll builds (e.g., Server 2019 17763.x) for splunk-winevtlog.exe?
- What VC++ Redistributable version is required/recommended for UF 10.0 (x64/x86)? Any compatibility matrix?
- Any known bugs/hotfixes for these crashes in UF 10.0?

Note: Problematic hosts seem to have older runtime builds than the working ones. Thanks for any pointers/docs!

[Mike_Prest1]

We're seeing the same crashes and it started with 9.4.5. 10.0.0 didn't help and either did 10.0.1. Still crashing.

[kiran_panchavat]

@krynol 

try disabling Security resolution (evt_resolve_ad_obj = 0)

etc/apps/Splunk_TA_windows/local/inputs.conf

[WinEventLog://Security]
evt_resolve_ad_obj = 0

evt_resolve_ad_obj = <boolean>
* How the input should interact with Active Directory while indexing Windows
  Event Log events.
* If you set this setting to true, the input resolves the Active
  Directory Security IDentifier (SID) objects to their canonical names for
  a specific Windows Event Log channel.
* If you enable the setting, the rate at which the input reads events
  on high-traffic Event Log channels can decrease. Latency can also increase
  during event acquisition. This is due to the overhead involved in performing
  AD translations.
* When you set this setting to true, you can optionally specify the domain
  controller name or dns name of the domain to bind to with the 'evt_dc_name'
  setting. The input connects to that domain controller to resolve the AD
  objects.
* If you set this setting to false, the input does not attempt any resolution.
* Default: false (disabled) for all channels

Please check this documentation https://splunk.my.site.com/customer/s/article/High-CPU-and-Memory-Usage-After-Splunk-UF-10-Upgrade 

[Andre_]

is that fixed in 9.4.7?

[Andre_]

in our case that solved the issue,

interesting side note - only UFs where high level of cross AD domain resolution happens were impacted. 

[krynol]

Thanks, but in my case, this doesn't solve the problem.

The only thing that helped was downgrading to version 9.4.3, and the errors disappeared. After I reinstalled 10.0.1, the problem no longer occurred.