Getting Data In

Powershell Input: discrepancies in pipeline execution?


I have a simple powershell input:

script = Get-WmiObject -Class Win32_Product | Select-Object -Property Name,InstallDate,Version,InstallLocation,Description

We deploy this input inside the Splunk_TA_Windows app to our Windows workstations - roughly 1800+ of them, a mix of Windows 7 and Windows 10. This input works flawlessly to report installed applications across all the workstations (after setting reasonable powershell execution policy via GPO, of course).

However, roughly 8 Windows 10 hosts (which are a mix of 15063 and 16299), all running the 7.0 forwarder, seem to be executing this powershell input differently.

We log Powershell event logs from WinEventLog:Microsoft-Windows-PowerShell/Operational, and had not yet filtered out Splunk Powershell events - I was doing some basic review of these event logs when I noticed a higher sustained volume EventCode 4103 from a handful of hosts, which led to the discovery that the forwarder on these 8 hosts appears to be running powershell pipeline executions *roughly 900 times a minute. *

The events differ from the "normal/expected" operation because in the Powershell event logs, they invoke the "Out-Null" method, followed by a "Sleep-Start" method (for 200ms). On every other host not exhibiting this behavior, the forwarder executes with the "Add-Type" method.

Regardless of this difference, the problematic hosts still execute the powershell - they just create a TON more noise doing it... I'm curious if this is a bug in the forwarder, or some possible difference in the powershell environment for these hosts.

Things I've done to troubleshoot:
- Reinstalled forwarders (no change)
- Upgraded forwarders to 7.0.3 (no change)
- tried using schedule instead of interval directive in the input (no change)

For example - problematic host has 4103 for splunk-powershell.exe that starts with:
CommandInvocation(Out-Null): "Out-Null"

These are followed by a nearly identical 4103, except with:
CommandInvocation(Start-Sleep): "Start-Sleep"
ParameterBinding(Start-Sleep): name="Milliseconds"; value="200"

Where a host not exhibiting the 900 events a minute starts like this:
CommandInvocation(Add-Type): "Add-Type"
ParameterBinding(Add-Type): name="MemberDefinition"; value="[DllImport("kernel32.dll")]
public static extern bool SetEvent(IntPtr handle);"
ParameterBinding(Add-Type): name="Name"; value="Win32Utils"
ParameterBinding(Add-Type): name="Namespace"; value="SetEvent"
ParameterBinding(Add-Type): name="PassThru"; value="True"

Hoping someone has some ideas as I'd like to address this instead of just drop the noise at our heavy forwarder..

Labels (1)
Tags (1)

Path Finder

Did you solve this issue?

0 Karma


I never did. I ended up filtering these events out via props/transforms on a heavy forwarder. I haven't revisited since we've updated our forwarders over the past year or two. Are you experiencing this as well? What version of the forwarder?

0 Karma

Path Finder

Thanks for the reply c.boggs.

I have just updated to 8.0.3. Admittedly I haven't used script stanza's before so unsure what to expect or if it affected our fleet.

All the scripts run flawlessly through a PowerShell CLI, but not all function when executed by Splunk.
Sysmon doesnt seem to generate many Process Creation events so at least that is one consolation.

    Pipeline execution details for command line:             start-sleep -m 200

    Context Information: 


        HostApplication=powershell.exe -command & {get-content C:\WINDOWS\TEMP\\input78ad6966241a2009.tmp | C:\Program` Files\SplunkUniversalForwarder\bin\splunk-powershell.ps1  C:\Program` Files\SplunkUniversalForwarder  78ad6966241a2009}
        CommandLine=            start-sleep -m 200

    CommandInvocation(Start-Sleep): "Start-Sleep"
    ParameterBinding(Start-Sleep): name="Milliseconds"; value="200"
0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...