Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)?

[jasondell]

This is the route we are heading:

[perfmon://ProcessandProcessor]
object = Process.*
counters = % Processor Time;ID Process
instances = *
index=perfmon
disabled=0
interval=30
whitelist=winlogon

winlogon is just an example.

The number of processes we need to monitor depend on the number of users logging into the server. So it could be 1 or 40 of the same process. The instances of the process have the # and a number based on its instance. How do we use regex or something like it to capture the CPU information for only those instances?

[JDukeSplunk · Accepted solution]

This might not be exactly what you were looking for, but I use WMI:LocalProcess to do this.

Example in the inputs.conf.

## Processes
[WMI:LocalProcesses]
interval = 120
wql = Select IDProcess,PrivateBytes,Name,PercentProcessorTime,TimeStamp_Sys100NS from Win32_PerfRawData_PerfProc_Process
index = idx_appdev
disabled = 0

And the search used to parse the info. In this search, I'm looking for process named "chrome*" and how much processor time it uses.

index=idx_appdev chrome* Name!=_Total Name!=Idle 
| reverse 
| streamstats current=f last(PercentProcessorTime) as last_PercentProcessorTime last(Timestamp_Sys100NS) as last_Timestamp_Sys100NS by Name 
| eval cputime = 100 * (PercentProcessorTime - last_PercentProcessorTime) / (Timestamp_Sys100NS - last_Timestamp_Sys100NS)
| search cputime > 0 AND cputime < 400
|  timechart span=1m avg(cputime) by Name useother=f limit=40

If all your looking for is the number of these, you could add your process name keyword in and..

 index=idx_appdev chrome* Name!=_Total Name!=Idle |timechart span=2m count(Name) by host

[jasondell]

We are going the WMI route because the data format is better. The only thing I did differently was add a where clause to the WQL to filter down to the exact process string.

[JDukeSplunk]

Glad it worked out for you. In that case, here is my entire wmi.conf. If you want it.

# WMI FOR appdev INDEX
#replace the index = line with the correct index 
#place this file in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local

[settings]
initial_backoff = 5
max_backoff = 20
max_retries_at_max_backoff = 0
checkpoint_sync_interval = 2

## Processes
[WMI:LocalProcesses]
interval = 120
wql = Select IDProcess,PrivateBytes,Name,PercentProcessorTime,TimeStamp_Sys100NS from Win32_PerfRawData_PerfProc_Process
index = idx_appdev
disabled = 0


## Scheduled Jobs

## Use the Win32_ScheduledJob  class. Note that this class can only return jobs that are created using either a script or AT.exe. 
## It cannot return information about jobs that are either created by or modified by the Scheduled Task wizard.
[WMI:ScheduledJobs]
disabled = 0
## Run once per day
interval = 86400
wql = SELECT Caption, Command, Description, InstallDate, InteractWithDesktop, JobId, JobStatus, Name, Notify, Priority, RunRepeatedly, Status FROM Win32_ScheduledJob
index = idx_appdev

## Services

## http://msdn.microsoft.com/en-us/library/aa394418(VS.85).aspx
## Lists all services registered on the system,if they are running,and the status
[WMI:Service]
disabled = 0
## Run once an hour
interval = 3600
wql = SELECT Name, Caption, State, Status, StartMode, StartName, PathName, Description FROM Win32_Service
index = idx_appdev


## Update
[WMI:InstalledUpdates]
disabled = 0
## Run once per day
interval = 86400
wql = SELECT Description, FixComments, HotFixID, InstalledBy, InstalledOn, ServicePackInEffect FROM Win32_QuickFixEngineering
index = idx_appdev


## Uptime
[WMI:Uptime]
disabled = 0
## Run once an hour
interval = 3600
wql = SELECT SystemUpTime FROM Win32_PerfFormattedData_PerfOS_System
index = idx_appdev

## index = idx_appdev


## Version
[WMI:Version]
disabled = 0
## Run once per day
interval = 86400
wql = SELECT Caption, ServicePackMajorVersion, ServicePackMinorVersion, Version FROM Win32_OperatingSystem
index = idx_appdev

And also the half-baked dashboard we use.

<form>
  <label>WMI Dashboard</label>
  <fieldset submitButton="true" autoRun="false">
    <input type="multiselect" token="hostname" searchWhenChanged="false">
      <label>Host Group</label>
      <search>
        <query>index=* sourcetype="WMI:Service" |dedup host |eval host=upper(host) |search host=* |sort -host</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <choice value="stwweb01 OR host=stwweb02 OR host=stwweb03 OR host=stwweb04">*STWWEB Prod</choice>
      <choice value="swf4d* host!=swf4d*q host!=swf4d*d">*4D Production</choice>
      <choice value="stwmt*">*STWMT</choice>
      <prefix>(</prefix>
      <suffix>)</suffix>
      <fieldForLabel>host</fieldForLabel>
      <fieldForValue>host</fieldForValue>
      <valuePrefix>host=</valuePrefix>
      <delimiter> OR </delimiter>
      <default>stwweb01 OR host=stwweb02 OR host=stwweb03 OR host=stwweb04</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <title>Summary Table</title>
        <searchString>index=* $hostname$  (sourcetype="WMI:InstalledUpdates" OR sourcetype="WMI:Uptime" OR sourcetype="WMI:Version")  | eventstats dc(HotFixID) as "Number of Patches" by host   | eval DaysUp=round(SystemUpTime/60/60/24,2)  |eventstats latest(DaysUp) as "Uptime" by host   | where sourcetype="WMI:Version"  |rex "Caption=(?&lt;OS&gt;.*)"  |stats latest(Uptime) as Uptime latest(OS) as OS latest(Version) as Version latest(ServicePackMajorVersion) as SP_Major# latest(ServicePackMinorVersion) as SP_Minor#  latest("Number of Patches") as "Number of Patches" by host |sort + host</searchString>
        <earliestTime>-36h</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">25</option>
      </table>
    </panel>
    <panel>
      <table>
        <title>Number of Patches (Sometimes breaks on summary table)</title>
        <searchString>$hostname$  (index=idx_appdev) sourcetype="WMI:InstalledUpdates" |stats dc(HotFixID) as "Number of Patches" by host</searchString>
        <earliestTime>-26h</earliestTime>
        <latestTime>now</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">false</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">column</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">30</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <title>OS Version</title>
        <searchString>index=* $hostname$ sourcetype="WMI:Version"   | dedup 1 host  | rex "Caption=(?&lt;OS&gt;.*)"  | eval OSVersion=OS."-SP ".ServicePackMajorVersion| top limit=20 OSVersion</searchString>
        <earliestTime>-24h@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">false</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">pie</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
      </chart>
    </panel>
    <panel>
      <chart>
        <title>7 Day Uptime Graph</title>
        <searchString>index=idx_appdev $hostname$ sourcetype="WMI:*" sourcetype="WMI:Uptime"  | eval DaysUp=round(SystemUpTime/60/60/24,2)| timechart span=1h avg(DaysUp) as Uptime by host useother=f limit=20</searchString>
        <earliestTime>-7d@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">false</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.nullValueMode">connect</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>Last WindowsUpdate (Work in progress)</title>
        <searchString>index=* $hostname$  sourcetype="WMI:InstalledUpdates" | eval epochtime=strptime(InstalledOn,"%m/%d/%Y")| eval Updated=strftime(epochtime,"20%y-%m-%d") |sort +Updated |stats max(Updated) as "Last Updates Installed" by host |sort + host</searchString>
        <earliestTime>-24h@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">20</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>Login Events</title>
        <search>
          <query>index=* $hostname$  index=idx_security sourcetype="WinEventLog:Security" Keywords="Audit Success"  (Account_Name=* NOT "ANONYMOUS LOGON" NOT svc* NOT *$ NOT - NOT IUSR_DATSTAT NOT SYSTEM NOT DefaultAppPool NOT webservice.external.weighttalkweb.com) (Security_ID=* NOT CHP\svc* NOT WEB\svc*) |eval LoginType=case(Logon_Type=3,"RPC",Logon_Type=4,"Batch",Logon_Type=5,"Service",Logon_Type=7,"Unlock",Logon_Type=10,"RDP/Terminal",Logon_Type=11,"Cached",Logon_Type=9,"New Credentials") |stats count(Account_Name) as "Login/Off Events" by Account_Name LoginType, host |sort + Account_Name</query>
          <earliest>-4h</earliest>
          <latest>now</latest>
        </search>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">10</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>KB installed by Host</title>
        <searchString>index=idx_appdev $hostname$ sourcetype="WMI:InstalledUpdates" | rex "Description=(?&lt;Type&gt;.*)" |stats dc(host) as "Hosts Installed On" by HotFixID Type |sort -"Hosts Installed On"</searchString>
        <earliestTime>-7d@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">20</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>By Update Type</title>
        <searchString>index=idx_appdev $hostname$ sourcetype="WMI:InstalledUpdates" | rex "Description=(?&lt;Update_Type&gt;.*)" |stats dc(HotFixID) as "Number" by Update_Type |sort - "Number"</searchString>
        <earliestTime>-24h@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">20</option>
      </table>
    </panel>
    <panel>
      <table>
        <title>Services Running</title>
        <searchString>index=idx_appdev $hostname$ sourcetype="WMI:Service" | rex "Caption=(?&lt;Name&gt;.*)"|stats dc(host) as "Hosts" by Name |sort + Hosts</searchString>
        <earliestTime>-4h@m</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">20</option>
      </table>
    </panel>
  </row>
</form>