WinEventLog:* on Windows 2008 and Splunk 4.3.2 forwarders - Splunk could not get the description for this event.

[jeff]

We have Universal Forwarders installed on Windows 2003 & 2008 Servers, plus a heavy forwarder on Windows 2008...

We updated to 4.3.2 on all forwarders in April, and converted all but one system configured as heavy forwarders to universal forwarders. Most of the systems were previously running 4.2.4 heavy forwarders, though a few were running 4.3.1 Universal Forwarders.

Last week I noticed, 11 of my 15 Windows forwarders displayed the "Splunk could not get the description for this event" message in 4,647 events for a 24 hour period, excluding domain controller security logs (in which case it goes into the millions). In the case of the domain controller, cycling the SplunkForwarder service once or twice usually clears up the messages from the WinEventLog:Security, though I'll continue to get the error message on the DCs in the Application and System Logs.

05/08/2012 01:19:29 PM LogName=System SourceName=Service Control Manager EventCode=7040 EventType=4 ComputerName=DC2.hersheymed.net User=SYSTEM Sid=S-1-5-18 SidType=1 TaskCategory=None OpCode=None RecordNumber=211980 Keywords=None Message=Splunk could not get the description for this event. Either the component that raises this event is not installed on your local computer or the installation is corrupt. FormatMessage error: The handle is invalid. Got the following information from this event: Windows Modules Installer demand start auto start TrustedInstaller

All 11 are Windows 2008(32-bit, 64-bit, and R2), the other four are all Windows 2003. The number of messages in the System and Application logs that display this behavior far exceeds the number of messages that do not. Indexes are 4.3.2 on RedHat, in case it matters. There are no (or very few if they're buried in the data) events with this behavior prior to updating the forwarder on any given host.

I've had a support case open since late last week, but I thought I'd ask the community if they can think of anything to check while I'm waiting... we're continuing to pull in corrupt (well, incomplete anyway) log data from these Windows forwarders so the delay in the back-and-forth-by-email isn't appealing.

[marksnelling]

I'm also getting a lot of these messages since upgrading my Windows Universal Forwarders to 4.3.2. I have a number of Universal Forwarders running on Windows 2008 R2 all forwarding to a single 4.3.2 indexer running on Linux.

[rovechkin_splun]

Can you enable splunk logging? In etc\log.cfg set the following flags to DEBUG
category.WinEventLogAgent=DEBUG

category.WinEventLogInputProcessor=DEBUG

category.WinEventLogChannel=DEBUG

category.WinEventLog=DEBUG

restart splunk and look into var\splunkd.log for possible errors

[jeff]

Yeah, I had done that for support... generates a lot of messages like the following:

DEBUG WinEventLogChannel - formatMessageByFlag: EvtFormatMessage returned no message, flag='(7)EvtFormatMessageProvider', channel='System', 'The handle is invalid.' WinEventLogChannel - getEventsNew: Failed to format source name of event log, channel='System', rec_id=434438 'The handle is invalid.'

several times for each event it couldn't process, but other than that not much useful. I downgraded to 4.3.1 on one forwarder yesterday and so far no errors...