I have a simple powershell input:
script = Get-WmiObject -Class Win32_Product | Select-Object -Property Name,InstallDate,Version,InstallLocation,Description
We deploy this input inside the Splunk_TA_Windows app to our Windows workstations - roughly 1800+ of them, a mix of Windows 7 and Windows 10. This input works flawlessly to report installed applications across all the workstations (after setting reasonable powershell execution policy via GPO, of course).
However, roughly 8 Windows 10 hosts (which are a mix of 15063 and 16299), all running the 7.0 forwarder, seem to be executing this powershell input differently.
We log Powershell event logs from WinEventLog:Microsoft-Windows-PowerShell/Operational , and had not yet filtered out Splunk Powershell events - I was doing some basic review of these event logs when I noticed a higher sustained volume EventCode 4103 from a handful of hosts, which led to the discovery that the forwarder on these 8 hosts appears to be running powershell pipeline executions *roughly 900 times a minute. *
The events differ from the "normal/expected" operation because in the Powershell event logs, they invoke the "Out-Null" method, followed by a "Sleep-Start" method (for 200ms). On every other host not exhibiting this behavior, the forwarder executes with the "Add-Type" method.
Regardless of this difference, the problematic hosts still execute the powershell - they just create a TON more noise doing it... I'm curious if this is a bug in the forwarder, or some possible difference in the powershell environment for these hosts.
Things I've done to troubleshoot:
- Reinstalled forwarders (no change)
- Upgraded forwarders to 7.0.3 (no change)
- tried using schedule instead of interval directive in the input (no change)
For example - problematic host has 4103 for splunk-powershell.exe that starts with:
These are followed by a nearly identical 4103, except with:
ParameterBinding(Start-Sleep): name="Milliseconds"; value="200"
Where a host not exhibiting the 900 events a minute starts like this:
ParameterBinding(Add-Type): name="MemberDefinition"; value="[DllImport("kernel32.dll")]
public static extern bool SetEvent(IntPtr handle);"
ParameterBinding(Add-Type): name="Name"; value="Win32Utils"
ParameterBinding(Add-Type): name="Namespace"; value="SetEvent"
ParameterBinding(Add-Type): name="PassThru"; value="True"
Hoping someone has some ideas as I'd like to address this instead of just drop the noise at our heavy forwarder..
... View more