We have Universal Forwarders installed on Windows 2003 & 2008 Servers, plus a heavy forwarder on Windows 2008...
We updated to 4.3.2 on all forwarders in April, and converted all but one system configured as heavy forwarders to universal forwarders. Most of the systems were previously running 4.2.4 heavy forwarders, though a few were running 4.3.1 Universal Forwarders.
Last week I noticed, 11 of my 15 Windows forwarders displayed the "Splunk could not get the description for this event" message in 4,647 events for a 24 hour period, excluding domain controller security logs (in which case it goes into the millions). In the case of the domain controller, cycling the SplunkForwarder service once or twice usually clears up the messages from the WinEventLog:Security, though I'll continue to get the error message on the DCs in the Application and System Logs.
05/08/2012 01:19:29 PM
LogName=System
SourceName=Service Control Manager
EventCode=7040
EventType=4
ComputerName=DC2.hersheymed.net
User=SYSTEM
Sid=S-1-5-18
SidType=1
TaskCategory=None
OpCode=None
RecordNumber=211980
Keywords=None
Message=Splunk could not get the description for this event. Either the component that raises this event is not installed on your local computer or the installation is corrupt.
FormatMessage error: The handle is invalid.
Got the following information from this event:
Windows Modules Installer
demand start
auto start
TrustedInstaller
All 11 are Windows 2008(32-bit, 64-bit, and R2), the other four are all Windows 2003. The number of messages in the System and Application logs that display this behavior far exceeds the number of messages that do not. Indexes are 4.3.2 on RedHat, in case it matters. There are no (or very few if they're buried in the data) events with this behavior prior to updating the forwarder on any given host.
I've had a support case open since late last week, but I thought I'd ask the community if they can think of anything to check while I'm waiting... we're continuing to pull in corrupt (well, incomplete anyway) log data from these Windows forwarders so the delay in the back-and-forth-by-email isn't appealing.
... View more