Splunk Heavy forwarder is able to connect to windows server via WMI but failed to collect WMI:WinEventLog:Security

[daniel_splunk]

Windows event from WMI:WinEventLog:System and WMI:WinEventLog:Application is fine. The only problem is failed to collect WMI:WinEventLog:Security,

As it can collect Application and System Windows event, connection is not a problem.

How can I debug further on this?

[daniel_splunk]

I enabled the following DEBUG for WMI.

$SPLUNK_HOME/etc/log.cfg
[splunkd]
category.ExecProcessor=DEBUG

$SPLUNK_HOME/etc/log-cmdlog.cfg
category.WMI=DEBUG

Then, I got the following from splunkd.log

01-03-2018 17:03:31.713 +0800 DEBUG ExecProcessor - message from ""E:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - getEventLogWql: DESC: chk=4294967295, low=110155, hi=4294967295 (10.13.18.59: Security)

01-03-2018 17:03:31.713 +0800 DEBUG ExecProcessor - message from ""E:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Event log wql "SELECT Category, CategoryString, ComputerName, EventCode, EventIdentifier, EventType, Logfile, Message, RecordNumber, SourceName, TimeGenerated, TimeWritten, Type, User FROM Win32_NTLogEvent WHERE Logfile = "Security" AND RecordNumber > 110155" (10.13.18.59: Security)

01-03-2018 17:03:31.713 +0800 DEBUG ExecProcessor - message from ""E:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Executing query wql="SELECT Category, CategoryString, ComputerName, EventCode, EventIdentifier, EventType, Logfile, Message, RecordNumber, SourceName, TimeGenerated, TimeWritten, Type, User FROM Win32_NTLogEvent WHERE Logfile = "Security" AND RecordNumber > 110155" (10.13.18.59: Security)

From the above debug log, the message for Security event shows the rec-id hits the limit of unsigned int type:

01-03-2018 17:03:31.713 +0800 DEBUG ExecProcessor - message from ""E:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - getEventLogWql: DESC: chk=4294967295, low=110155, hi=4294967295 (10.13.18.59: Security)

The limit of unsigned int is 4294967295 (0xffffffff).

Here's a link for the limitation of Microsoft WQL api:
https://social.technet.microsoft.com/Forums/windowsserver/en-US/78e6d555-0f5d-4def-92d5-14d3ad6ee558...

As stated in the link, the rec ids are limited to 32 bit unsigned int. WMI does not work if the rec id goes beyond that point.

You can try to configure event logs to smaller sizes so that the logs rotate before rec id hitting the limit.