Powershell Input: discrepancies in pipeline execution?

[c_boggs]

I have a simple powershell input:

[powershell://Windows:Applications]
script = Get-WmiObject -Class Win32_Product | Select-Object -Property Name,InstallDate,Version,InstallLocation,Description
interval=86400
disabled=0
index=winclientlogs
sourcetype=installed_apps

We deploy this input inside the Splunk_TA_Windows app to our Windows workstations - roughly 1800+ of them, a mix of Windows 7 and Windows 10. This input works flawlessly to report installed applications across all the workstations (after setting reasonable powershell execution policy via GPO, of course).

However, roughly 8 Windows 10 hosts (which are a mix of 15063 and 16299), all running the 7.0 forwarder, seem to be executing this powershell input differently.

We log Powershell event logs from WinEventLog:Microsoft-Windows-PowerShell/Operational, and had not yet filtered out Splunk Powershell events - I was doing some basic review of these event logs when I noticed a higher sustained volume EventCode 4103 from a handful of hosts, which led to the discovery that the forwarder on these 8 hosts appears to be running powershell pipeline executions *roughly 900 times a minute. *

The events differ from the "normal/expected" operation because in the Powershell event logs, they invoke the "Out-Null" method, followed by a "Sleep-Start" method (for 200ms). On every other host not exhibiting this behavior, the forwarder executes with the "Add-Type" method.

Regardless of this difference, the problematic hosts still execute the powershell - they just create a TON more noise doing it... I'm curious if this is a bug in the forwarder, or some possible difference in the powershell environment for these hosts.

Things I've done to troubleshoot:
- Reinstalled forwarders (no change)
- Upgraded forwarders to 7.0.3 (no change)
- tried using schedule instead of interval directive in the input (no change)

For example - problematic host has 4103 for splunk-powershell.exe that starts with:
CommandInvocation(Out-Null): "Out-Null"

These are followed by a nearly identical 4103, except with:
CommandInvocation(Start-Sleep): "Start-Sleep"
ParameterBinding(Start-Sleep): name="Milliseconds"; value="200"

Where a host not exhibiting the 900 events a minute starts like this:
CommandInvocation(Add-Type): "Add-Type"
ParameterBinding(Add-Type): name="MemberDefinition"; value="[DllImport("kernel32.dll")]
public static extern bool SetEvent(IntPtr handle);"
ParameterBinding(Add-Type): name="Name"; value="Win32Utils"
ParameterBinding(Add-Type): name="Namespace"; value="SetEvent"
ParameterBinding(Add-Type): name="PassThru"; value="True"

Hoping someone has some ideas as I'd like to address this instead of just drop the noise at our heavy forwarder..

[mik3y]

Did you solve this issue?

[c_boggs]

I never did. I ended up filtering these events out via props/transforms on a heavy forwarder. I haven't revisited since we've updated our forwarders over the past year or two. Are you experiencing this as well? What version of the forwarder?

[mik3y]

Thanks for the reply c.boggs.

I have just updated to 8.0.3. Admittedly I haven't used script stanza's before so unsure what to expect or if it affected our 7.3.1.1 fleet.

All the scripts run flawlessly through a PowerShell CLI, but not all function when executed by Splunk.
Sysmon doesnt seem to generate many Process Creation events so at least that is one consolation.

    Pipeline execution details for command line:             start-sleep -m 200
    . 

    Context Information: 
        DetailSequence=1
        DetailTotal=1

        SequenceNumber=570469

        UserId=DOMAIN\SYSTEM
        HostName=ConsoleHost
        HostVersion=5.1.14409.1018
        HostId=9db3f578-72c9-4efe-8ec5-2987f958b4a0
        HostApplication=powershell.exe -command & {get-content C:\WINDOWS\TEMP\\input78ad6966241a2009.tmp | C:\Program` Files\SplunkUniversalForwarder\bin\splunk-powershell.ps1  C:\Program` Files\SplunkUniversalForwarder  78ad6966241a2009}
        EngineVersion=5.1.14409.1018
        RunspaceId=91737811-5a9e-430a-9462-cdb540f6e006
        PipelineId=1
        ScriptName=
        CommandLine=            start-sleep -m 200


    Details: 
    CommandInvocation(Start-Sleep): "Start-Sleep"
    ParameterBinding(Start-Sleep): name="Milliseconds"; value="200"