All Apps and Add-ons

PowerShell Modular input doesn't process my sourcetype for the data.

Communicator

I had a scripted input with power-shell as simply, *.bat files pointing to *.ps1 files and I was able to use my sourcetype by inputs.conf and props.conf.
We recently upgraded our system to Splunk 6.3 and decided to user PowerShell Modular input from UI since we would have a chance to edit schedules without restarting system etc..
Although, it is the same same power-shell script and props.conf, Splunk indexer fails to set sourcetype to my predefined type although I pick my source-type from the list. Instead, each time I save from UI as "from the list", whenever I go back to modular input page, it says manual and simply breaking my event into line by line.
Please advise as this is a bug for Splunk PowerShell modular input or I am missing something in the process?
Thanks upfront for your time.

0 Karma
1 Solution

SplunkTrust
SplunkTrust

According to the documentation you should be formatting your output differently when using in conjunction with modular input and powershell app.

http://docs.splunk.com/Documentation/AddOns/latest/MSPowerShell/Writescriptsforthemodularinput#Outpu...

Important: The modular input currently requires that any PowerShell scripts it executes produce output objects that do not have any script properties. Pipe output through | Select-Object * to ensure proper formatting.

 $Directories = @("c:\test","c:\windows")
 $now = $(get-date).ToString()

 foreach ($item in $Directories){
         $directoryInfo = $(Get-ChildItem $item -ErrorAction silentlycontinue| Measure-Object).Count
         $howManyDirectories = $(Get-ChildItem $item -ErrorAction silentlycontinue | where {$_.PSIsContainer} | Measure-Object).Count
         $howManyFiles = $(Get-ChildItem c:\ -ErrorAction silentlycontinue | where {$_.mode -notlike "d*"} | Measure-Object).Count
         if(!(Test-Path -Path $item )){
             write-output( $now + " no directory " + $item) | Select-Object *

         }
         elseif($directoryInfo -eq $directoryInfo){
             write-output($now + " " +$item + $onlyDirectory) | Select-Object *
             #I really dont understand this elseif and it's purpose  I have dir with one sub dir and it always gets stuck here.
         }
         else {
             $date = (Get-Date).AddHours(-4)
             write-output $((get-childitem $item | where-object {$_.LastWriteTime -lt $date -and !$_.PSIsContainer}| foreach-object { Write-Output $now " File: " $_.FullName " LastWriteTime: " $_.LastWriteTime | Select-Object *}))

         }
 }

You'll note I scratched your echos and same error handling variables you had and I dont understand the elseif. I think its meant to be if the directory is empty. I made you variables in your foreach equal to numbers instead of arrays with numbers attached which simplifies their usage later. Biggest point to get across here is the usage of the |select-object *

View solution in original post

SplunkTrust
SplunkTrust

According to the documentation you should be formatting your output differently when using in conjunction with modular input and powershell app.

http://docs.splunk.com/Documentation/AddOns/latest/MSPowerShell/Writescriptsforthemodularinput#Outpu...

Important: The modular input currently requires that any PowerShell scripts it executes produce output objects that do not have any script properties. Pipe output through | Select-Object * to ensure proper formatting.

 $Directories = @("c:\test","c:\windows")
 $now = $(get-date).ToString()

 foreach ($item in $Directories){
         $directoryInfo = $(Get-ChildItem $item -ErrorAction silentlycontinue| Measure-Object).Count
         $howManyDirectories = $(Get-ChildItem $item -ErrorAction silentlycontinue | where {$_.PSIsContainer} | Measure-Object).Count
         $howManyFiles = $(Get-ChildItem c:\ -ErrorAction silentlycontinue | where {$_.mode -notlike "d*"} | Measure-Object).Count
         if(!(Test-Path -Path $item )){
             write-output( $now + " no directory " + $item) | Select-Object *

         }
         elseif($directoryInfo -eq $directoryInfo){
             write-output($now + " " +$item + $onlyDirectory) | Select-Object *
             #I really dont understand this elseif and it's purpose  I have dir with one sub dir and it always gets stuck here.
         }
         else {
             $date = (Get-Date).AddHours(-4)
             write-output $((get-childitem $item | where-object {$_.LastWriteTime -lt $date -and !$_.PSIsContainer}| foreach-object { Write-Output $now " File: " $_.FullName " LastWriteTime: " $_.LastWriteTime | Select-Object *}))

         }
 }

You'll note I scratched your echos and same error handling variables you had and I dont understand the elseif. I think its meant to be if the directory is empty. I made you variables in your foreach equal to numbers instead of arrays with numbers attached which simplifies their usage later. Biggest point to get across here is the usage of the |select-object *

View solution in original post

Communicator

in Response to jkat54:
Assuming I have index "X" and sourcetype "Y", following code is in for each loop in the script:

$date = (Get-Date).AddHours(-4)
echo((get-childitem $item  *.* | where-object {$_.LastWriteTime -lt $date -and !$_.PSIsContainer}| %{ Write-Output $now " File: " $_.FullName " LastWriteTime: " $_.LastWriteTime}))

how do i suppose to override sourcetype or index? Why I can't set this up from Splunk side.
Script works perfect with .bat and .ps1 combination. It picks up the sourcetype i desire on Splunk.

0 Karma

SplunkTrust
SplunkTrust

Hello, some things have changed with relation to powershell scripts in version 6.3. The batch files are no longer needed, and there are some additional configuration items. Please see this link for more details:

http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/MonitorWindowsDatawithPowerShellscripts

0 Karma

SplunkTrust
SplunkTrust

Example of using write-host alias (echo) vs using write-host:

alt text

0 Karma

Communicator

thanks for the answer. I checked the link and couldn't really identify anything particularly good for my question except overriding Splunk properties in powershell. I added code sample from my PS script in my question. Could you take a look and tell me what would make it work? any addition to powershell code?

0 Karma

SplunkTrust
SplunkTrust

The echo makes no sense to me. $item isnt defined, $now isnt defined, string variable concatenation didnt look right.

How about this:

 $date = (Get-Date).AddHours(-4)
 get-childitem $item  *.* | where-object {$_.LastWriteTime -lt $now -and !$_.PSIsContainer}| foreach-object { $msg= '$(get-date) + " File: " + $_.FullName + "  LastWriteTime:   " +  $_.LastWriteTime'; write-output $msg}

may need write-host instead... i dont know

0 Karma

Communicator

I feel more like something buggy on Splunk side. Same script works perfectly with .bat file but not with Modular input :S

0 Karma

SplunkTrust
SplunkTrust

Show me your batch script too please. And the full powershell.

I'll explain why one works and other doesn't.

0 Karma

Communicator

Man with batch script, ps1 works perfectly. Problem is ps1 only modular input doesn't pick the sourcetype. Here is full foreach loop:

foreach ($item in $Directories){
        #echo ( $now + " " +$item)
        $directoryInfo = Get-ChildItem $item -ErrorAction silentlycontinue| Measure-Object
        $howManyDirectories = Get-ChildItem $item -ErrorAction silentlycontinue | where {$_.PSIsContainer} | Measure-Object
        $howManyFiles = Get-ChildItem $item -ErrorAction silentlycontinue | where { $_.PSIsLeaf} | Measure-Object
        #dir $item -ErrorAction silentlycontinue
        if(!(Test-Path -Path $item )){
            echo ( $now + " " +$noDirectory + $item)

        }
        elseif($directoryInfo.count -eq 0){
            echo($now + " " +$noFileFound + $item)
        }elseif($howManyDirectories.Count -eq $directoryInfo.Count){
            echo( $now + " " +$item + $onlyDirectory)
        }
        else{
            $date = (Get-Date).AddHours(-4)
            #echo($now + " Directory:  " +
            #$item +
            echo((get-childitem $item  *.* | where-object {$_.LastWriteTime -lt $date -and !$_.PSIsContainer}| %{ Write-Output $now " File: " $_.FullName " LastWriteTime: " $_.LastWriteTime}))
            #echo dir

        }
}
0 Karma

SplunkTrust
SplunkTrust

I believe the way you execute as batch is affecting the way you write your powershell. And so when you run your powershell using the cmd prompt and the batch file... it produces appropriate output, whereas when you run just the powershell from powershell, its not producing the expected results.

Try running it both ways and comparing the outputs.

0 Karma

SplunkTrust
SplunkTrust

If you find this to be true, then show me your batch and your powershell and I'll tell you how to re-write your powershell script so that it can stand on its own without command line interpreter overhead.

0 Karma