How to monitor 150+ instances of the same service in perfmon

mmqt — Mon, 15 Jul 2019 16:15:34 GMT

So my systems can spawn upto and above 150+ instances of the same application. I'm using the generic perfmon Process monitor:

[perfmon://Process]
counters = % Processor Time; % User Time; % Privileged Time; Virtual Bytes Peak; Virtual Bytes; Page Faults/sec; Working Set Peak; Working Set; Page File Bytes Peak; Page File Bytes; Private Bytes; Thread Count; ID Process; Pool Paged Bytes; Pool Nonpaged Bytes; Handle Count; IO Read Operations/sec; IO Write Operations/sec; IO Data Operations/sec; IO Other Operations/sec; IO Read Bytes/sec; IO Write Bytes/sec; IO Data Bytes/sec; IO Other Bytes/sec; Working Set - Private
disabled = 0
instances = myapp
interval = 1
mode = single
object = Process
useEnglishOnly=true
index = perfmon

My concern is that instances can't use wildcards in naming standards.
instances = myapp* doesn't work. Only works when doing instances = * which I don't want. I could write a script that generates myapp#1,myapp#2,myapp#3 (etc.) but I'm worried that's not going to be the best way to deploy this monitor.

Does anyone have any other suggestions?
Thanks

Re: How to monitor 150+ instances of the same service in perfmon

mmqt — Mon, 22 Jul 2019 23:38:08 GMT

If anyone stumbles into this question in the future, I wasn't able to solve the initial problem of monitoring my processes with perfmon so I setup a powershell script and a custom app, my app is configured as follows

local/inputs.conf

# Process Monitor script
[script://.\bin\myapp.path]
interval = 10
disabled = 0

#monitor output of proc
[monitor://$SPLUNK_HOME\var\log\myapp\proc.csv]
disabled = 0
sourcetype = myappProc
interval = 10
crcSalt = <SOURCE>
index = oswin

local/props.conf

[myappProc]
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n]+)
FIELD_DELIMITER = ,
FIELD_NAMES = Name,StartTime,cpu_user_percent,NPM,PM,WS(MB),WS,VM,PID,Path,user
FIELD_QUOTE = "
INDEXED_EXTRACTIONS = csv
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
description = myapp Process Monitor
disabled = false
pulldown_type = true

bin/myapp.path

$SystemRoot\System32\WindowsPowerShell\v1.0\powershell.exe -command  " & 'C:\Program Files\SplunkUniversalForwarder\etc\apps\myapp\bin\myappproc.ps1'"

bin/myappproc.ps1

$CPUPercent = @{
  Name = 'CPU'
  Expression = {
    $TotalSec = (New-TimeSpan -Start $_.StartTime).TotalSeconds
    [Math]::Round( ($_.CPU * 100 / $TotalSec), 3)
  }
}
$owners = @{}
gwmi win32_process |% {$owners[$_.handle] = $_.getowner().user}

Set-Variable -Name "LogFolder" -Value "C:\Program Files\SplunkUniversalForwarder\var\log\myapp"
Set-Variable -Name "MonitoredLogFile" -Value "C:\Program Files\SplunkUniversalForwarder\var\log\myapp\proc.csv"

if (!(Test-Path -Path $LogFolder )) {
        New-Item -ItemType directory -Path $LogFolder
}


$Processes = Get-Process |
Where-Object -property Path -like "*MYAPP*"|
 Select-Object -Property Name,StartTime, $CPUPercent,NPM,PM,{$_.WorkingSet /1mb},WS,VM,Id,Path,@{l="Owner";e={$owners[$_.id.tostring()]}} |
 Select-Object
 $output = ForEach ($Process in $Processes){
        $Process
        }
$output |ConvertTo-Csv -NoTypeInformation |Select-Object -Skip 1| Set-Content -Path $MonitoredLogFile

Hope this helps anyone who finds this.

topic How to monitor 150+ instances of the same service in perfmon in Splunk Search

How to monitor 150+ instances of the same service in perfmon

Re: How to monitor 150+ instances of the same service in perfmon