I managed to set up my WMI event-polling setup and it mostly works.
Mostly, because it doesn't pull events from non-standard event logs like - for example - the WMI log itself.
I know that in order to be able to see the event log via WMI I have to add an entry to registy (in my case it's Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\Microsoft-Windows-WMI-Activity/Operational). I did that.
And I can list the events with wbemtest as per https://docs.splunk.com/Documentation/Splunk/8.2.2/Troubleshooting/TroubleshootingWMI
I can also do:
splunk cmd splunk-wmi.exe -wql "SELECT Category, CategoryString, ComputerName, EventCode, EventIdentifier, EventType, Logfile, Message, RecordNumber, SourceName, TimeGenerated, TimeWritten, Type, User FROM Win32_NTLogEvent WHERE Logfile = \"Microsoft-Windows-WMI-Activity/Operational\"" -namespace \\ad.lab\root\cimv2
And it works (returns events).
But If I set
event_log_file = System, Security, Application, Microsoft-Windows-WMI-Activity/Operational
in my wmi.conf file, only the "standard" log events are getting pulled (System, Security and Application).
splunkd.log doesn't show anything regarding wmi apart from
09-21-2021 09:29:34.002 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - Running: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe" on PipelineSet 0
09-21-2021 09:29:34.002 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - PipelineSet 0: Created new ExecedCommandPipe for ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"", uniqueId=1
09-21-2021 09:29:34.221 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Attempting to connect to WMI provider \\ad.lab\root\cimv2
09-21-2021 09:29:34.267 +0200 INFO ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Connected to WMI provider \\ad.lab\root\cimv2 (connecting took 46.84 milliseconds)
09-21-2021 09:29:34.267 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Attempting to connect to WMI provider \\ad.lab\root\cimv2
09-21-2021 09:29:34.267 +0200 INFO ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Connected to WMI provider \\ad.lab\root\cimv2 (connecting took 0 microseconds)
09-21-2021 09:29:34.267 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Attempting to connect to WMI provider \\ad.lab\root\cimv2
09-21-2021 09:29:34.283 +0200 INFO ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Connected to WMI provider \\ad.lab\root\cimv2 (connecting took 15.62 milliseconds)
09-21-2021 09:29:34.283 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Attempting to connect to WMI provider \\ad.lab\root\cimv2
09-21-2021 09:29:34.283 +0200 INFO ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Connected to WMI provider \\ad.lab\root\cimv2 (connecting took 0 microseconds)
09-21-2021 09:29:34.502 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Executing query wql="SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA "Win32_NTLogEvent" AND TargetInstance.Logfile = "System"" (ad.lab: System)
09-21-2021 09:29:34.502 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Executing query wql="SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA "Win32_NTLogEvent" AND TargetInstance.Logfile = "Security"" (ad.lab: Security)
09-21-2021 09:29:34.502 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Executing query wql="SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA "Win32_NTLogEvent" AND TargetInstance.Logfile = "Application"" (ad.lab: Application)
09-21-2021 09:29:34.502 +0200 DEBUG ExecProcessor [7720 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Executing query wql="SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA "Win32_NTLogEvent" AND TargetInstance.Logfile = "Microsoft-Windows-WMI-Activity/Operational"" (ad.lab: Microsoft-Windows-WMI-Activity/Operational)
What more can I debug?
The same goes even if I want to query my local (the one where I have the UF installed) machine with WMI. "Normal" logs get pulled, the Services and Application ones - don't. If I set them up as normal wineventlog source, they get pulled correctly.
EDIT: OK, I don't know how but it just started working on its own. I have no idea what changed. I didn't do anything on the remote machine, I only added the splunk UF user to DCOM users on the local machine and tried pulling different logs from both machines. For now it seems it's working with WMI both localy, and from remote server.