Ehhh, I'm trying to set up polling for remote events using WMI (yes, I know it's easier to install UF on the destination machine but I can't do it in this case).
I know the docs (https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/MonitorWMIdata) and I'm trying to do as it says. In the lab for now, so some permissions might be overly loose.
I created a domain account gave it full local admin rights on the Splunk UF machine, I installed UF to run with this account. I installed the TA for windows.
Pointed the UF to the indexer - first success - the _internal log is filling with logs from the forwarder. So the connectivity works.
Added proper Security Policies through GPO, added DCOM group membership, added WMI namespace security permissions to \\root and \\root\cimv2 for my UF domain user.
Added wmi.conf stanza for remote WMI. And I'm getting
09-20-2021 16:53:44.622 +0200 ERROR ExecProcessor [8152 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Executing query failed (query="SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA "Win32_NTLogEvent" AND TargetInstance.Logfile = "Security"") (error="Current user does not have permission to perform the action." HRESULT=80041003) (ad.lab: Security)
If I do wbemtest, as described in https://docs.splunk.com/Documentation/Splunk/8.2.2/Troubleshooting/TroubleshootingWMI I can connect and authenticate properly to the remote server. But if I do the test query I get empty results.
Of course the logs on the queried server's side are very helpful - say that there is an error and "cause=unknown".
Id = {00000000-0000-0000-0000-000000000000}; ClientMachine = SEP; User = LAB\splunkuf; ClientProcessId = 5860; Component = Unknown; Operation = Start IWbemServices::ExecNotificationQuery - root\cimv2 : SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA "Win32_NTLogEvent" AND TargetInstance.Logfile = "Security"; ResultCode = 0x80041032; PossibleCause = Unknown
Debugging on UF's side also isn't very helpful
09-20-2021 17:57:28.845 +0200 DEBUG ExecProcessor [3796 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Attempting to connect to WMI provider \\ad.lab\root\cimv2
09-20-2021 17:57:28.845 +0200 INFO ExecProcessor [3796 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Connected to WMI provider \\ad.lab\root\cimv2 (connecting took 0 microseconds)
09-20-2021 17:57:28.939 +0200 DEBUG ExecProcessor [3796 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Executing query wql="SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA "Win32_NTLogEvent" AND TargetInstance.Logfile = "Application"" (ad.lab: Application)
09-20-2021 17:57:28.939 +0200 ERROR ExecProcessor [3796 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Executing query failed (query="SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA "Win32_NTLogEvent" AND TargetInstance.Logfile = "Application"") (error="Current user does not have permission to perform the action." HRESULT=80041003) (ad.lab: Application)
09-20-2021 17:57:28.939 +0200 INFO ExecProcessor [3796 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"" WMI - Will retry connection to WMI provider after 5.000 seconds (ad.lab: Application)
As tou can see, I can connect but the WMI query fails.
And now I don't know whether it's the case of WMI-level permissions, some other permissions? I hope I don't have to add additional permissions to event logs because that's ridiculous and no sane administrator will let me anter SDDL's directly into registry.
If I run the event viewer on the splunk UF machine as the splunk UF domain user, I can connect to the server I want to monitor, but I cannot open any logs. It says "Event Viewer cannot open the event log or custom view. Verify that Event Log service is runnint or query is too long. The operation completed successfully. (5)"
Any hints where to look for help?
OK. It seems the issue was very besic although as a mainly non-windows user I easily overlooked it (and the docs don't say a word about it).
The user that the UF is running with must be a member of Event Log Readers group on the destination server.
I'm still struggling with reading Applications and Services logs but that's another story.
OK. It seems the issue was very besic although as a mainly non-windows user I easily overlooked it (and the docs don't say a word about it).
The user that the UF is running with must be a member of Event Log Readers group on the destination server.
I'm still struggling with reading Applications and Services logs but that's another story.