How to resolve crash events in windows application log for App: splunk-winevtlog.exe (eventcode = 1000)?

[mortf]

I'm having som issues with the application log on some of our windows servers getting spammed with the following messages:

Faulting application name: splunk-winevtlog.exe, version: 1794.768.23581.39240, time stamp: 0x5c1d9d74
Faulting module name: KERNELBASE.dll, version: 6.3.9600.19724, time stamp: 0x5ec5262a
Exception code: 0xeeab5254
Fault offset: 0x0000000000007afc
Faulting process id: 0x3258
Faulting application start time: 0x01d787a1d9f141cd
Faulting application path: C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 18687572-f395-11eb-8131-005056b32672
Faulting package full name: 
Faulting package-relative application ID: 

Always followed by a 1001 information event like so:

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: splunk-winevtlog.exe
P2: 1794.768.23581.39240
P3: 5c1d9d74
P4: KERNELBASE.dll
P5: 6.3.9600.19724
P6: 5ec5262a
P7: eeab5254
P8: 0000000000007afc
P9: 
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_splunk-winevtlog_32b957db7bcb27fbdcdd5be64aea86e1b639666_0170a0ed_a993dd7e

Analysis symbol: 
Rechecking for solution: 0
Report Id: 18687572-f395-11eb-8131-005056b32672
Report Status: 4100
Hashed bucket:  

I've tried a lot of changes to the Universal Forwarder configuration but nothing i do removes these message. The only thing i've noticed that can helt to remove these messages is by lowering the memory consumption on the server. So far the servers i've seen with these message in the application log are running at 70% and more memory consumption. But 70% memory consumption seems to be normal and i don't see why this should cause the splunk-winevtlog.exe to crash (as often as every minute).  

Our version of Splunk Universal Forwarder is 7.2.3. I've checked the "known issues" on splunk docs but can't fint anything related to memory issues for this version.

I'm thinking about upgrading the Universal Forwarder to a newer version, but that's just because i can't think og anything else to try. Do anyone else experience this and know what can be done?

As a side note: Splunk internal shows absolutely nothing. There are no warnings or errors at all in the internal log on these servers. But the event spamming (crashes) are still logged in the windows application log. Splunk itself does not log or detect a crash it seems?

[nunoaragao]

Did you check the values of Report_Id I mentioned earlier @mdsnmss ? Are they repeating or all unique?

[KaraD]

Hi @aleccese -- Looks like this question has gained a lot of interest, but it is originally from 2021. Could you please post this issue as a new question to gain more visibility?

Thanks!

-Kara

Splunk Community Manager

[mykol_j]

I think this is all a "non issue" as it relates to Splunk.... read on:  if you search for this error as posted by the OP, you will see faults generated by Splunk. To illustrate, use this ugly query (don't do this at home, this for demonstration purposes only):

index=*wineventlog "faulting" "*splunk*"

HOWEVER, take off the "*splunk*" from this, and you'll see that hundreds of other apps are doing this too.

I'm calling this a Windows issue, not a Splunk issue. 🙄

FWIW:  version 9.1.1.

[aleccese]

Sorry but this is not my case. I have no other application faulting in my application event viewer, only splunk-winevtlog.exe is crashing. Moreover if I disable the [WinEventLog://ForwardedEvents] input it stops crashing, meaning that there is something related to it (maybe the amount of events, maybe the renderXml phase, I don't know).

We first set ForwardedEvents size to 8 GB in EventViewer, I believed this was the issue so I tried with 256 MB but it still crashes.

I also tried to remove any whitelist/blacklist instruction from inputs (I read there is a know issue about that) with no luck.

When the process crashes I got this on splunkd.log:

10-06-2023 11:42:39.214 +0200 WARN TcpOutputProc [6564 indexerPipe] - Pipeline data does not have indexKey. [_path] = C:\Program Files\Splunk\bin\splunk-winevtlog.exe\n[_raw] = \n[_meta] = punct::\n[_stmid] = tMwn7kh87nMs0iK\n[MetaData:Source] = source::WinEventLog\n[MetaData:Host] = host::PRD-LOGCOLL-SRV\n[MetaData:Sourcetype] = sourcetype::WinEventLog\n[_done] = _done\n[_linebreaker] = _linebreaker\n[_conf] = source::WinEventLog|host::PRD-LOGCOLL-SRV|WinEventLog|\n

I would be more than glad to confirm that it's a "non issue" but in my case events are not collected or seem to be collected partially with huge delays.

Regards

Alessandro

[mykol_j]

wow, definitely a case of "your mileage may differ"... this is just a small sample of these alerts here, and on the ones triggered by Splunk, they still seem to function OK:

ADPClientService.exe, version: 4.1.38.0, time stamp: 0x62c69205
AUEPMaster.exe, version: 1910.24.6.725, time stamp: 0x5d39726f
AdAutoUpdateSDK.dll, version: 0.0.0.0, time stamp: 0x61dc3463
AdskAccessServiceHost.exe, version: 1.27.0.4, time stamp: 0x61dc35ae
AdskUpdateCheck.exe, version: 1.27.0.4, time stamp: 0x61dc3558
CcmProfiler.dll_unloaded, version: 5.0.9106.1000, time stamp: 0x642d9f3d
FMEngine.dll, version: 19.2.2.234, time stamp: 0x60451558
KERNEL32.DLL, version: 10.0.17763.4720, time stamp: 0xa2ec4df3
KERNELBASE.dll, version: 10.0.19041.3393, time stamp: 0x6b4de7c9
OUTLOOK.EXE, version: 16.0.10402.20023, time stamp: 0x64ef06a7
smartscreenps.dll, version: 10.0.19041.3031, time stamp: 0x92650ce8
PDFMEngine.dll, version: 23.6.20320.0, time stamp: 0x64f8d26b
RPCRT4.dll, version: 10.0.17763.4644, time stamp: 0x565f63ab
RtkAudUService64.exe, version: 1.0.0.176, time stamp: 0x5c6f93ad
VCRUNTIME140.dll, version: 14.16.27033.0, time stamp: 0x5d30eadf
biwinrt.dll, version: 10.0.17763.2989, time stamp: 0x790cc0bc

splunk-winevtlog.exe, version: 2304.1280.25713.15594, time stamp: 0x64713ec1

[nunoaragao]

Oh! If the issue is tied toWinEventLog://ForwardedEvents maybe you'd like to also comment on splunk-winevtlog.exe keeps crashing post. I also stumbled of this input stanza on my investigation, but on our estate, we're not enabling it.

[nunoaragao]

.. but that is your own post. sorry.