Getting Data In

Why is powershell scripted Input via universal forwarder not being indexed?

wsgr_mccurity
Loves-to-Learn Lots

The below setup doesn't appear to index the script's output and I can't figure out why.  Even the basic one-liner example in their documentation (https://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowsdatawithPowerShellscripts) doesn't produce indexed events for me.  I've tried several variations on how the data is being formatted.  I know the script executes because the file change it makes is occurring.

configureBINDIP.ps1

$launchConfFile = "C:\Program Files\SplunkUniversalForwarder\etc\splunk-launch.conf"
$launchConfSetting = "SPLUNK_BINDIP=127.0.0.1"

function CraftEvent ($message) {
$event = [PSCustomObject]@{
"SplunkIndex" = "windows"
"SplunkSource" = "powershell"
"SplunkSourceType" = "Powershell:ConfigureBINDIP"
"SplunkHost" = "mysplunkhost"
"SplunkTime" = (New-TimeSpan -Start $(Get-Date -Date "01/01/1970") -End $(Get-Date)).TotalSeconds
"Message" = $message
}

Return $event
}

if (-not (Test-Path $launchConfFile) ) {
$event = [PSCustomObject]@{
"Message" = "Could not locate splunk-launch.conf: $launchConfFile"
}
Write-Output $event | Select-Object
exit
}

if ( (Get-Content $launchConfFile ) -notcontains $launchConfSetting ) {
$message = "Appending '$launchConfSetting' to '$launchConfFile'"
"`r`n$launchConfSetting" | Out-File $launchConfFile -Append utf8

if ( (Get-Content $launchConfFile ) -contains $launchConfSetting ) {
$message += ".... splunk-launch.conf update successful. Please remove this host from the app to restart."
} else {
$message += ".... splunk-launch.conf does not appear updated. Please continue to monitor."
}
} else {
$message = "splunk-launch.conf already appears updated. Please remove this host from the app to restart."
}

$event = [PSCustomObject]@{
"Message" = $message
}

Write-Output $event | Select-Object

inputs.conf

[powershell://ConfigureBINDIP]
script = . "$SplunkHome\etc\apps\configure_bindip\bin\configureBINDIP.ps1"
index = windows
source = powershell
sourcetype = Powershell:ConfigureBINDIP

 web.conf

[settings]
mgmtHostPort = 127.0.0.1:8089

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You can check your _internal logs from that forwarder whether any events were produced.

Something like that

index=_internal sourcetype=splunkd host=<your_forwarder> component=Metrics group=per_source_thruput series=powershell*
0 Karma

wsgr_mccurity
Loves-to-Learn Lots

I don't see any events with the powershell* series.

EDIT - I see them for splunk-powershell*.  FWIW to help expedite testing I cranked up the schedule to every minute, the metrics in this log represent that and not the default "run once" schedule.

 

06-13-2022 11:32:08.790 -0700 INFO  Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.014, eps=0.097, kb=0.419, ev=3, avg_age=0.000, max_age=0
06-13-2022 11:31:06.774 -0700 INFO  Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.014, eps=0.097, kb=0.420, ev=3, avg_age=0.000, max_age=0
06-13-2022 11:30:04.767 -0700 INFO  Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.014, eps=0.097, kb=0.420, ev=3, avg_age=20.000, max_age=60
06-13-2022 11:29:33.765 -0700 INFO  Metrics - group=per_sourcetype_thruput, series="splunk-powershell.ps-2", kbps=0.000, eps=0.032, kb=0.000, ev=1, avg_age=0.000, max_age=0

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Hmm... indeed seems that your script does generate events.

Do you generate timestamp with your events? You might be hitting the "no timestamp so splunk assigns last event's timestamp" problem. This way all events end up at the same time in the past.

0 Karma

wsgr_mccurity
Loves-to-Learn Lots

I've tried adding a timestamp to the PSObject and also without adding one (note the function in the above script is not ever called but is a good example of how I'd be adding that timestamp). 

The index they're (supposed to be) going to doesn't show the event even if searching all time.  That index does not regularly receive events so there's very little data there.  A real-time search on that index also does not show the events as they're arriving.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

I checked my script and the main part of it boils down to this:

[...]
foreach ($inputline in [...])
{
   $output=@{}
   #Manipulate external data, extract some fields, set $output.fields
[...]
   Write-Output $output
}

From what I found in the trimmed out block:

 # First we need a timestamp
# Edit: No, we don't. We get the timestamp from the forwarder when the script is run
# $output.time =( Get-Date -Format "dd-mm-yyyy HH:mm:ss K" )

As you can see - I tried to set time manually but resigned in the end and rely on the forwarder supplying proper timestamp at ingest time.

For me it works.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Check the splunk-powershell.log

The script might be starting but failing in the middle somewhere.

0 Karma

wsgr_mccurity
Loves-to-Learn Lots

I don't see anything unusual in there.  Only three entries per run: Queued, Start execution, and End execution.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...