topic Re: Powershell Input: discrepancies in pipeline execution? in Getting Data In

Powershell Input: discrepancies in pipeline execution?

c_boggs — Tue, 29 Sep 2020 18:59:43 GMT

I have a simple powershell input:
[powershell://Windows:Applications] script = Get-WmiObject -Class Win32_Product | Select-Object -Property Name,InstallDate,Version,InstallLocation,Description interval=86400 disabled=0 index=winclientlogs sourcetype=installed_apps
We deploy this input inside the Splunk_TA_Windows app to our Windows workstations - roughly 1800+ of them, a mix of Windows 7 and Windows 10. This input works flawlessly to report installed applications across all the workstations (after setting reasonable powershell execution policy via GPO, of course).

However, roughly 8 Windows 10 hosts (which are a mix of 15063 and 16299), all running the 7.0 forwarder, seem to be executing this powershell input differently.

We log Powershell event logs from WinEventLog:Microsoft-Windows-PowerShell/Operational, and had not yet filtered out Splunk Powershell events - I was doing some basic review of these event logs when I noticed a higher sustained volume EventCode 4103 from a handful of hosts, which led to the discovery that the forwarder on these 8 hosts appears to be running powershell pipeline executions *roughly 900 times a minute. *

The events differ from the "normal/expected" operation because in the Powershell event logs, they invoke the "Out-Null" method, followed by a "Sleep-Start" method (for 200ms). On every other host not exhibiting this behavior, the forwarder executes with the "Add-Type" method.

Regardless of this difference, the problematic hosts still execute the powershell - they just create a TON more noise doing it... I'm curious if this is a bug in the forwarder, or some possible difference in the powershell environment for these hosts.

Things I've done to troubleshoot:
- Reinstalled forwarders (no change)
- Upgraded forwarders to 7.0.3 (no change)
- tried using schedule instead of interval directive in the input (no change)

For example - problematic host has 4103 for splunk-powershell.exe that starts with:
CommandInvocation(Out-Null): "Out-Null"

These are followed by a nearly identical 4103, except with:
CommandInvocation(Start-Sleep): "Start-Sleep" ParameterBinding(Start-Sleep): name="Milliseconds"; value="200"

Where a host not exhibiting the 900 events a minute starts like this:
CommandInvocation(Add-Type): "Add-Type" ParameterBinding(Add-Type): name="MemberDefinition"; value="[DllImport("kernel32.dll")] public static extern bool SetEvent(IntPtr handle);" ParameterBinding(Add-Type): name="Name"; value="Win32Utils" ParameterBinding(Add-Type): name="Namespace"; value="SetEvent" ParameterBinding(Add-Type): name="PassThru"; value="True"

Hoping someone has some ideas as I'd like to address this instead of just drop the noise at our heavy forwarder..

Re: Powershell Input: discrepancies in pipeline execution?

mik3y — Wed, 03 Jun 2020 05:02:10 GMT

Did you solve this issue?

Re: Powershell Input: discrepancies in pipeline execution?

c_boggs — Wed, 03 Jun 2020 18:30:34 GMT

I never did. I ended up filtering these events out via props/transforms on a heavy forwarder. I haven't revisited since we've updated our forwarders over the past year or two. Are you experiencing this as well? What version of the forwarder?

Re: Powershell Input: discrepancies in pipeline execution?

mik3y — Wed, 03 Jun 2020 23:33:05 GMT

Thanks for the reply c.boggs.

I have just updated to 8.0.3. Admittedly I haven't used script stanza's before so unsure what to expect or if it affected our 7.3.1.1 fleet.

All the scripts run flawlessly through a PowerShell CLI, but not all function when executed by Splunk.
Sysmon doesnt seem to generate many Process Creation events so at least that is one consolation.

    Pipeline execution details for command line:             start-sleep -m 200
    . 

    Context Information: 
        DetailSequence=1
        DetailTotal=1

        SequenceNumber=570469

        UserId=DOMAIN\SYSTEM
        HostName=ConsoleHost
        HostVersion=5.1.14409.1018
        HostId=9db3f578-72c9-4efe-8ec5-2987f958b4a0
        HostApplication=powershell.exe -command & {get-content C:\WINDOWS\TEMP\\input78ad6966241a2009.tmp | C:\Program` Files\SplunkUniversalForwarder\bin\splunk-powershell.ps1  C:\Program` Files\SplunkUniversalForwarder  78ad6966241a2009}
        EngineVersion=5.1.14409.1018
        RunspaceId=91737811-5a9e-430a-9462-cdb540f6e006
        PipelineId=1
        ScriptName=
        CommandLine=            start-sleep -m 200


    Details: 
    CommandInvocation(Start-Sleep): "Start-Sleep"
    ParameterBinding(Start-Sleep): name="Milliseconds"; value="200"