Why "FormatMessage error" appears in indexed message for Windows security event logs?

[mgaraventa_splu]

Since a while the Message field of my Windows security event logs is not extracted properly and in Splunk I see the Message field having following value instead:

Message=Splunk could not get the description for this event. Either the component that raises this event is not installed on your local computer or the installation is corrupt. FormatMessage error:

If I restart the server, it doesn't help and the issue keeps on reoccurring. How can I fix this issue? Thanks in advance for your help.

[jijulukose]

Wanted to share some good news regarding this issue with the broader community - over the past year we worked with Splunk and Microsoft support and development teams to sort this out.

In our organization, 99% of the corrupt events were when Windows was rebooting - a fix for this was identified and successfully tested in our environment.

Remaining issue was related to an api error response from EventLog api and a fix for that is in the works per Splunk Dev and support.

Both fixes are expected to make it into 9.1 (hopefully around .CONF23).

And here's some recommendations that came from the collaborative work with Microsoft.

1. Use 'Delayed Start' for the Splunk Forwarder service.
2. Setup following service dependencies to reduce errors
   1. sc config EventLog depend=RpcSs
   2. sc config SplunkForwarder depend=EventLog

[jbillings]

Same issue on UF 7.2.0 Indexer 7.3.1

[atsichlis]

I am also seeing this.

[dagar_ruralking]

Did you ever get this resolved? I am seeing this as well.

[mayler]

Same issue. UF is 7.1 and Indexer is 7.1.1. Opening Case.

[dagar_ruralking]

Did you ever get this resolved? I am seeing this as well.

[mgaraventa_splu · Accepted solution]

Splunk Engineering believes this was introduced in a Microsoft Windows patch. The workaround is to configure a delayed start of the Splunk service(s) so that it starts after the Windows Event Log service, as explained earlier.

[danielbb]

---

@mgaraventa_splu wrote:

Splunk Engineering believes this was introduced in a Microsoft Windows patch. The workaround is to configure a delayed start of the Splunk service(s) so that it starts after the Windows Event Log service, as explained earlier.

How can we configure such a delay?

---

[xiyangyang]

does this happen to ver6.4.5 UF too?

[LyDang]

This is also happening after a delayed start. Also happening after restarting the Splunk UF service.

[mgaraventa_splu]

As a first step we need to make sure that it's not an issue of a missing dll or possibly an issue with the event format.

I. first let's check if the necessary dll is present on the splunk instance responsible for the Message field resolution:

• go to registry key HKEY_LOCAL_MACHINE\\SYSTEM\CurrentControlSet\services\eventlog.
• find the event source you need and then the right subkey (for instance Microsoft-Windows-Security-Auditing) in the message. The EventMessageFile contains %SystemRoot%\system32\adtschema.dll. That is the DLL we need on the forwarder host.
• if missing, look for that dll on a different server of the same kind, copy the dll over, even changing the path if you feel. Then you need to setup the same keys in the registry to point to that DLL. The easiest way is to export the key from the original server and import on the forwarder host, eventually changing the dll paths.
• reboot the forwarder host.
• monitor if the issue is still happening.

II. if the issue is not the dll or if the issue should persist also after fixing issue number 1, we need to make sure that event format is not the issue here:

• list the subscriptions on your collector by issuing wecutil enum-subscription
• change the event format from RenderedText (default) to Events: wecutil ss /cf:Events
• monitor if the issue is still happening.

III. if the issue is not the event format or if the issue should persist also after fixing issues number 1 and 2, then we might be facing a new issue, still under investigation, for which no fix has been identified yet, but which is usually workarounded successfully in following way:

• try if a splunk restart solves the issue
• if yes, then the workaround is to configure a delayed start of the splunk service(s) so that it starts after the Windows Event Log service. In fact it seems that the splunk service(s) starting before the Windows Event Log service is triggering this issue.

This last issue has been seen on different Splunk 6.1.x versions, mainly in Splunk 6.1.2, 6.1.3 and 6.1.4 and on 6.2.x versions (both UF and Splunk Enterprise) and on different Windows OSes (Windows 2008 Standard x86, Windows 2008 R2 Standard x64, Windows 2008 R2 Enterprise, Windows 2008 R2, Windows 2012 Standard x64, Windows 2012 R2). What has triggered this issue has not been clarified yet (some users tell that the issue began to occur after upgrading Splunk, other say that it began to occur after installing MS updates/patches).

For anyone interested in finding a proper solution for this issue, I would strongly recommend to file a new support case and to provide following pieces of information in order to help Splunk Support to get all necessary information to properly identify the root cause:

A. what exactly has changed on the host before the issue began to occur?
A.1. did you upgrade Splunk before seeing the issue on the affected hosts?
A.2. did you install any specific Windows updates on those boxes before the issue began to occur? If yes, could you please provide me the list of these updates?
B. could you please confirm which are the exact OS versions on which the affected splunk instances are running?
C. which are the exact Splunk versions affected by the issue?
D. a sample of the original affected Windows event log.
E. please enable DEBUG for the WinEventLogChannel processor. Please make sure that the log level is adjusted before the issue is reproduced, otherwise the logs will not have the necessary verbosity.
F. the output of the command splunk cmd splunkd print-modinput-config WinEventLog | splunk-WinEvtLog.exe >> "winevtlog.output". Before you execute that you need to make sure that the $SPLUNK_HOME variable is set properly, for example:

set SPLUNK_HOME="c:\program files\splunk"

winevtlog.output is an output file name of your choice. When you execute the command it will open a cmd window where some logs will be quickly displayed. Please keep the window open for sufficient long time in order to be sure that we capture a moment when the issue occurs. Afterwards please close the cmd window and this will dump everything to the output file specified.
G. a diag of the affected splunk instance.
H. please disable DEBUG for the WinEventLogChannel processor.