How to troubleshoot why Splunk stopped indexing WMI Windows Security Events?

[rbal_splunk]

Splunk stopped receiving Windows Security Event using WMI.
This has been working for year; also issue is only with security event all other channel are still being ingested using WMI.

[rbal_splunk · Accepted solution]

For such issues my Suggestion will be for you to enable following debugging on the instance of Splunk where these WMI dats is being ingested.
To enable tracing please use http://wiki.splunk.com/Community:TroubleshootingWMIIssues as suggested in the document you need to enable following followed by restart of splun kinstance.

In $SPLUNK_HOME/etc/log.cfg:
category.ExecProcessor=DEBUG

$SPLUNK_HOME/etc/logcmdline.cfg:
category.WMI=DEBUG

After that check if Splunkd.log shows message like rec-id hits the limit of unsigned int type:

05-15-2015 13:24:11.809 -0400 DEBUG ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - EventLogQueryProcessor::writeData: Reseting DESC: chk=4294967295, low=1033239786, hi=4294967295 (co1-dfsprd33.tampa.healthe: Security)

The limit of unsigned int is 4294967295 (0xffffffff).

Here's a link for the limitation of Microsoft WQL api:

https://social.technet.microsoft.com/Forums/windowsserver/en-US/78e6d555-0f5d-4def-92d5-14d3ad6ee558...

As stated in the link, the rec ids are limited to 32 bit unsigned int. WMI does not work if the rec id goes beyond that point.
Due to this limitation instead of using wmi suggestion will be to install Universal Forwarder at sources to ingest data that is currently ingested using WMI.

[m4him7]

The link that rbal_splunk gave seems to be broken. I found it here:
http://wiki.splunk.com/Community:TroubleshootingWMIIssues
although it seems to be the same link except a comma got stuck on the end of the link.

[peterchow]

Hi rbal,

For the wmi issue, is it possible to reset the record?

Thanks

Best REgards,
Peter