topic Re: Why is Powershell Scripted Input via Universal Forwarder Not being Indexed? in Getting Data In

Why is powershell scripted Input via universal forwarder not being indexed?

wsgr_mccurity — Thu, 30 Jun 2022 21:17:41 GMT

The below setup doesn't appear to index the script's output and I can't figure out why. Even the basic one-liner example in their documentation (https://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowsdatawithPowerShellscripts) doesn't produce indexed events for me. I've tried several variations on how the data is being formatted. I know the script executes because the file change it makes is occurring.

configureBINDIP.ps1

$launchConfFile = "C:\Program Files\SplunkUniversalForwarder\etc\splunk-launch.conf"
$launchConfSetting = "SPLUNK_BINDIP=127.0.0.1"

function CraftEvent ($message) {
  $event = [PSCustomObject]@{
    "SplunkIndex" = "windows"
    "SplunkSource" = "powershell"
    "SplunkSourceType" = "Powershell:ConfigureBINDIP"
    "SplunkHost" = "mysplunkhost"
    "SplunkTime" = (New-TimeSpan -Start $(Get-Date -Date "01/01/1970") -End $(Get-Date)).TotalSeconds
    "Message" = $message
  }

  Return $event
}

if (-not (Test-Path $launchConfFile) ) {
  $event = [PSCustomObject]@{
    "Message" = "Could not locate splunk-launch.conf: $launchConfFile"
  }
  Write-Output $event | Select-Object
  exit
}

if ( (Get-Content $launchConfFile ) -notcontains $launchConfSetting ) {
  $message = "Appending '$launchConfSetting' to '$launchConfFile'"
  "`r`n$launchConfSetting" | Out-File $launchConfFile -Append utf8

  if ( (Get-Content $launchConfFile ) -contains $launchConfSetting ) {
    $message += ".... splunk-launch.conf update successful. Please remove this host from the app to restart."
  } else {
    $message += ".... splunk-launch.conf does not appear updated. Please continue to monitor."
  }
} else {
  $message = "splunk-launch.conf already appears updated. Please remove this host from the app to restart."
}

$event = [PSCustomObject]@{
"Message" = $message
}

Write-Output $event | Select-Object

inputs.conf

[powershell://ConfigureBINDIP]
script = . "$SplunkHome\etc\apps\configure_bindip\bin\configureBINDIP.ps1"
index = windows
source = powershell
sourcetype = Powershell:ConfigureBINDIP

web.conf

[settings]
mgmtHostPort = 127.0.0.1:8089

Re: Powershell Scripted Input via Universal Forwarder Not being Indexed

PickleRick — Thu, 09 Jun 2022 06:08:16 GMT

Check the splunk-powershell.log

The script might be starting but failing in the middle somewhere.

Re: Powershell Scripted Input via Universal Forwarder Not being Indexed

wsgr_mccurity — Thu, 09 Jun 2022 13:11:59 GMT

I don't see anything unusual in there. Only three entries per run: Queued, Start execution, and End execution.

Re: Why is Powershell Scripted Input via Universal Forwarder Not being Indexed?

PickleRick — Mon, 13 Jun 2022 10:03:45 GMT

You can check your _internal logs from that forwarder whether any events were produced.

Something like that

index=_internal sourcetype=splunkd host=<your_forwarder> component=Metrics group=per_source_thruput series=powershell*

Re: Why is Powershell Scripted Input via Universal Forwarder Not being Indexed?

wsgr_mccurity — Mon, 13 Jun 2022 18:35:04 GMT

I don't see any events with the powershell* series.

EDIT - I see them for splunk-powershell*. FWIW to help expedite testing I cranked up the schedule to every minute, the metrics in this log represent that and not the default "run once" schedule.

06-13-2022 11:32:08.790 -0700 INFO Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.014, eps=0.097, kb=0.419, ev=3, avg_age=0.000, max_age=0 06-13-2022 11:31:06.774 -0700 INFO Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.014, eps=0.097, kb=0.420, ev=3, avg_age=0.000, max_age=0 06-13-2022 11:30:04.767 -0700 INFO Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.014, eps=0.097, kb=0.420, ev=3, avg_age=20.000, max_age=60 06-13-2022 11:29:33.765 -0700 INFO Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.000, eps=0.032, kb=0.000, ev=1, avg_age=0.000, max_age=0

Re: Why is Powershell Scripted Input via Universal Forwarder Not being Indexed?

PickleRick — Mon, 13 Jun 2022 19:37:11 GMT

Hmm... indeed seems that your script does generate events.

Do you generate timestamp with your events? You might be hitting the "no timestamp so splunk assigns last event's timestamp" problem. This way all events end up at the same time in the past.

Re: Why is Powershell Scripted Input via Universal Forwarder Not being Indexed?

wsgr_mccurity — Thu, 30 Jun 2022 20:25:14 GMT

I've tried adding a timestamp to the PSObject and also without adding one (note the function in the above script is not ever called but is a good example of how I'd be adding that timestamp).

The index they're (supposed to be) going to doesn't show the event even if searching all time. That index does not regularly receive events so there's very little data there. A real-time search on that index also does not show the events as they're arriving.

Re: Why is Powershell Scripted Input via Universal Forwarder Not being Indexed?

PickleRick — Fri, 01 Jul 2022 07:34:39 GMT

I checked my script and the main part of it boils down to this:

[...]
foreach ($inputline in [...])
{
   $output=@{}
   #Manipulate external data, extract some fields, set $output.fields
   [...]
   Write-Output $output
}

From what I found in the trimmed out block:

 # First we need a timestamp
# Edit: No, we don't. We get the timestamp from the forwarder when the script is run
# $output.time =( Get-Date -Format "dd-mm-yyyy HH:mm:ss K" )

As you can see - I tried to set time manually but resigned in the end and rely on the forwarder supplying proper timestamp at ingest time.

For me it works.