Getting Data In

WinEventLog:* on Windows 2008 and Splunk 4.3.2 forwarders - Splunk could not get the description for this event.

jeff
Contributor

We have Universal Forwarders installed on Windows 2003 & 2008 Servers, plus a heavy forwarder on Windows 2008...

We updated to 4.3.2 on all forwarders in April, and converted all but one system configured as heavy forwarders to universal forwarders. Most of the systems were previously running 4.2.4 heavy forwarders, though a few were running 4.3.1 Universal Forwarders.

Last week I noticed, 11 of my 15 Windows forwarders displayed the "Splunk could not get the description for this event" message in 4,647 events for a 24 hour period, excluding domain controller security logs (in which case it goes into the millions). In the case of the domain controller, cycling the SplunkForwarder service once or twice usually clears up the messages from the WinEventLog:Security, though I'll continue to get the error message on the DCs in the Application and System Logs.

05/08/2012 01:19:29 PM LogName=System SourceName=Service Control Manager EventCode=7040 EventType=4 ComputerName=DC2.hersheymed.net User=SYSTEM Sid=S-1-5-18 SidType=1 TaskCategory=None OpCode=None RecordNumber=211980 Keywords=None Message=Splunk could not get the description for this event. Either the component that raises this event is not installed on your local computer or the installation is corrupt. FormatMessage error: The handle is invalid. Got the following information from this event: Windows Modules Installer demand start auto start TrustedInstaller

All 11 are Windows 2008(32-bit, 64-bit, and R2), the other four are all Windows 2003. The number of messages in the System and Application logs that display this behavior far exceeds the number of messages that do not. Indexes are 4.3.2 on RedHat, in case it matters. There are no (or very few if they're buried in the data) events with this behavior prior to updating the forwarder on any given host.

I've had a support case open since late last week, but I thought I'd ask the community if they can think of anything to check while I'm waiting... we're continuing to pull in corrupt (well, incomplete anyway) log data from these Windows forwarders so the delay in the back-and-forth-by-email isn't appealing.

1 Solution

jrodman
Splunk Employee
Splunk Employee

Hm, I answered this, but the comment system ate it whole. Trying again.

This is indeed a bug in 4.3.2 WinEventLog data acquisition code. It's now been identified and we can fix for the next release.

The code flow was altered in order to address performance concerns, which was a success. The performance for rapidly acquiring eventlog data should improve by around a factor of 2, which relieves stress on the wineventlog service (which was the bottleneck). The changes have to do with cacheing handles for data providers of windows eventlog strings.

Of course that's a poor consolation for correct operation. Unfortunately, the set of events we tested with did not have the distribution of data providers, so the problem wasn't identified internally.

For now please use 4.3.1 forwarders (or earlier) to acquire this data type.

Please note that this error message does not have a one to one correlation with this misbehavior. Other scenarios such as loading EVT files without the corresponding availble DLLs that provide the messages, or reading eventlogs for an application which has been subsequently uninstalled could (and do) produce the same message.

View solution in original post

mship
Path Finder

Look at your windows event logs locally and I will bet you are getting the same message. If it is your security log you are probably missing the msaudite.dll file under system32 folder along with security subkey under the hklmsystemcurrentcontrolsetserviceseventlogsecurity.
If it is in the app or system event log you are missing the registry hives for those events. You can just copy them over from a working machine.

mship
Path Finder

Thank you Jeff but the above is not the only resolution as I have just worked through this same issue myself and it was not a bug with us as I rolled back to 4.3.1 and it did not fix the issue. What did fix the issue was that we were missing essential dll's and registry keys. I just wanted to provided more info for others where this solution may not be the answer. Thanks again and next time before giving negative points ensure that you have the proper knowledge.

0 Karma

jeff
Contributor

No, the event viewer view was fine looking at the same messages... it's a bug in 4.3.2 (SPL-51312 ) and is resolved in 4.3.3. You should read through the entire post - this was mentioned above.

0 Karma

gpt
New Member

Where can i get Universal forwarder 4.3.1 ??
Done!! Thank you!!! (Older versions link in the same download web ....)

Instead of the forwarder 4.3.1 the message of the event is not showed. Same message 'Splunk could not get ...'

Any idea?

0 Karma

jrodman
Splunk Employee
Splunk Employee

Hm, I answered this, but the comment system ate it whole. Trying again.

This is indeed a bug in 4.3.2 WinEventLog data acquisition code. It's now been identified and we can fix for the next release.

The code flow was altered in order to address performance concerns, which was a success. The performance for rapidly acquiring eventlog data should improve by around a factor of 2, which relieves stress on the wineventlog service (which was the bottleneck). The changes have to do with cacheing handles for data providers of windows eventlog strings.

Of course that's a poor consolation for correct operation. Unfortunately, the set of events we tested with did not have the distribution of data providers, so the problem wasn't identified internally.

For now please use 4.3.1 forwarders (or earlier) to acquire this data type.

Please note that this error message does not have a one to one correlation with this misbehavior. Other scenarios such as loading EVT files without the corresponding availble DLLs that provide the messages, or reading eventlogs for an application which has been subsequently uninstalled could (and do) produce the same message.

reedmohn
Communicator

I don't see this bug on the "know issues" list, and I don't see it on the list of things fixed in 4.3.3. Am I just not seeing it, or....?

0 Karma

mship
Path Finder

Rolling back to 4.3.1 UF did not work for me...any other suggestions?

0 Karma

jeff
Contributor

In May I was told 4.3.3 was targeted for July.

0 Karma

Lowell
Super Champion

For searching purposes, this issues is listed as SPL-51312.

0 Karma

Lowell
Super Champion

Any timeline for the release of 4.3.3?

0 Karma

bizza
Path Finder

I had the same issue, and now I'm downgrading the universal forwarder: hope it can solve the bug 🙂
Thank you

0 Karma

farren
New Member
0 Karma

coreindustries
New Member

Where can I download 4.3.1?

0 Karma

jrodman
Splunk Employee
Splunk Employee

I meant the message as it should have been gathered; for example as it appears with 4.3.1 or in event log viewer. I'm working on the bug -- and more information will help.
; well the problem turned out to have not much relationship with the message, so nevermind.

0 Karma

jeff
Contributor

You mean, other than the sample message in the post?... if you read through the rest of the post you'll see that I downgraded to 4.3.1 on key forwarders and the symptoms went away. Support is entering a bug report for me, as of late Friday afternoon.

0 Karma

jrodman
Splunk Employee
Splunk Employee

The likely relevant change was from 4.3.1 to 4.3.2; I would recommend using 4.3.1 for now. An example specific message which behaved that way would be useful.

0 Karma

the_wolverine
Champion

Have you considered downgrading the forwarder? We're not seeing this issue with UF 4.2.5.

0 Karma

the_wolverine
Champion

Our site has also encountered this. Smells like a bug as it has only occurred with the 4.3.2 UF, not with the previous version of UF (4.2.x)

0 Karma

JensT
Communicator

Hello,

what the SPL number for this?

Kind Regards,

Jens

0 Karma

jeff
Contributor

yeah, support is finally filing a bug report...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...