Why is powershell scripted Input via universal forwarder not being indexed?

[wsgr_mccurity]

The below setup doesn't appear to index the script's output and I can't figure out why.  Even the basic one-liner example in their documentation (https://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowsdatawithPowerShellscripts) doesn't produce indexed events for me.  I've tried several variations on how the data is being formatted.  I know the script executes because the file change it makes is occurring.

configureBINDIP.ps1

$launchConfFile = "C:\Program Files\SplunkUniversalForwarder\etc\splunk-launch.conf"
$launchConfSetting = "SPLUNK_BINDIP=127.0.0.1"

function CraftEvent ($message) {
  $event = [PSCustomObject]@{
    "SplunkIndex" = "windows"
    "SplunkSource" = "powershell"
    "SplunkSourceType" = "Powershell:ConfigureBINDIP"
    "SplunkHost" = "mysplunkhost"
    "SplunkTime" = (New-TimeSpan -Start $(Get-Date -Date "01/01/1970") -End $(Get-Date)).TotalSeconds
    "Message" = $message
  }

  Return $event
}

if (-not (Test-Path $launchConfFile) ) {
  $event = [PSCustomObject]@{
    "Message" = "Could not locate splunk-launch.conf: $launchConfFile"
  }
  Write-Output $event | Select-Object
  exit
}

if ( (Get-Content $launchConfFile ) -notcontains $launchConfSetting ) {
  $message = "Appending '$launchConfSetting' to '$launchConfFile'"
  "`r`n$launchConfSetting" | Out-File $launchConfFile -Append utf8

  if ( (Get-Content $launchConfFile ) -contains $launchConfSetting ) {
    $message += ".... splunk-launch.conf update successful. Please remove this host from the app to restart."
  } else {
    $message += ".... splunk-launch.conf does not appear updated. Please continue to monitor."
  }
} else {
  $message = "splunk-launch.conf already appears updated. Please remove this host from the app to restart."
}

$event = [PSCustomObject]@{
"Message" = $message
}

Write-Output $event | Select-Object

inputs.conf

[powershell://ConfigureBINDIP]
script = . "$SplunkHome\etc\apps\configure_bindip\bin\configureBINDIP.ps1"
index = windows
source = powershell
sourcetype = Powershell:ConfigureBINDIP

 web.conf

[settings]
mgmtHostPort = 127.0.0.1:8089

[PickleRick]

You can check your _internal logs from that forwarder whether any events were produced.

Something like that

index=_internal sourcetype=splunkd host=<your_forwarder> component=Metrics group=per_source_thruput series=powershell*

[wsgr_mccurity]

I don't see any events with the powershell* series.

EDIT - I see them for splunk-powershell*.  FWIW to help expedite testing I cranked up the schedule to every minute, the metrics in this log represent that and not the default "run once" schedule.

06-13-2022 11:32:08.790 -0700 INFO  Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.014, eps=0.097, kb=0.419, ev=3, avg_age=0.000, max_age=0
06-13-2022 11:31:06.774 -0700 INFO  Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.014, eps=0.097, kb=0.420, ev=3, avg_age=0.000, max_age=0
06-13-2022 11:30:04.767 -0700 INFO  Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.014, eps=0.097, kb=0.420, ev=3, avg_age=20.000, max_age=60
06-13-2022 11:29:33.765 -0700 INFO  Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.000, eps=0.032, kb=0.000, ev=1, avg_age=0.000, max_age=0

[PickleRick]

Hmm... indeed seems that your script does generate events.

Do you generate timestamp with your events? You might be hitting the "no timestamp so splunk assigns last event's timestamp" problem. This way all events end up at the same time in the past.

[wsgr_mccurity]

I've tried adding a timestamp to the PSObject and also without adding one (note the function in the above script is not ever called but is a good example of how I'd be adding that timestamp). 

The index they're (supposed to be) going to doesn't show the event even if searching all time.  That index does not regularly receive events so there's very little data there.  A real-time search on that index also does not show the events as they're arriving.

[PickleRick]

I checked my script and the main part of it boils down to this:

[...]
foreach ($inputline in [...])
{
   $output=@{}
   #Manipulate external data, extract some fields, set $output.fields
   [...]
   Write-Output $output
}

From what I found in the trimmed out block:

 # First we need a timestamp
# Edit: No, we don't. We get the timestamp from the forwarder when the script is run
# $output.time =( Get-Date -Format "dd-mm-yyyy HH:mm:ss K" )

As you can see - I tried to set time manually but resigned in the end and rely on the forwarder supplying proper timestamp at ingest time.

For me it works.

[PickleRick]

Check the splunk-powershell.log

The script might be starting but failing in the middle somewhere.

[wsgr_mccurity]

I don't see anything unusual in there.  Only three entries per run: Queued, Start execution, and End execution.