Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)?

jasondell — Thu, 16 Feb 2017 14:55:41 GMT

This is the route we are heading:

[perfmon://ProcessandProcessor]
object = Process.*
counters = % Processor Time;ID Process
instances = *
index=perfmon
disabled=0
interval=30
whitelist=winlogon

winlogon is just an example.

The number of processes we need to monitor depend on the number of users logging into the server. So it could be 1 or 40 of the same process. The instances of the process have the # and a number based on its instance. How do we use regex or something like it to capture the CPU information for only those instances?

Re: Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)?

JDukeSplunk — Thu, 16 Feb 2017 16:13:07 GMT

This might not be exactly what you were looking for, but I use WMI:LocalProcess to do this.

Example in the inputs.conf.

## Processes
[WMI:LocalProcesses]
interval = 120
wql = Select IDProcess,PrivateBytes,Name,PercentProcessorTime,TimeStamp_Sys100NS from Win32_PerfRawData_PerfProc_Process
index = idx_appdev
disabled = 0

And the search used to parse the info. In this search, I'm looking for process named "chrome*" and how much processor time it uses.

index=idx_appdev chrome* Name!=_Total Name!=Idle 
| reverse 
| streamstats current=f last(PercentProcessorTime) as last_PercentProcessorTime last(Timestamp_Sys100NS) as last_Timestamp_Sys100NS by Name 
| eval cputime = 100 * (PercentProcessorTime - last_PercentProcessorTime) / (Timestamp_Sys100NS - last_Timestamp_Sys100NS)
| search cputime > 0 AND cputime < 400
|  timechart span=1m avg(cputime) by Name useother=f limit=40

If all your looking for is the number of these, you could add your process name keyword in and..

 index=idx_appdev chrome* Name!=_Total Name!=Idle |timechart span=2m count(Name) by host

Re: Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)?

jasondell — Fri, 17 Feb 2017 18:52:51 GMT

We are going the WMI route because the data format is better. The only thing I did differently was add a where clause to the WQL to filter down to the exact process string.

Re: Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)?

JDukeSplunk — Fri, 17 Feb 2017 19:16:27 GMT

Glad it worked out for you. In that case, here is my entire wmi.conf. If you want it.

# WMI FOR appdev INDEX
#replace the index = line with the correct index 
#place this file in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local

[settings]
initial_backoff = 5
max_backoff = 20
max_retries_at_max_backoff = 0
checkpoint_sync_interval = 2

## Processes
[WMI:LocalProcesses]
interval = 120
wql = Select IDProcess,PrivateBytes,Name,PercentProcessorTime,TimeStamp_Sys100NS from Win32_PerfRawData_PerfProc_Process
index = idx_appdev
disabled = 0


## Scheduled Jobs

## Use the Win32_ScheduledJob  class. Note that this class can only return jobs that are created using either a script or AT.exe. 
## It cannot return information about jobs that are either created by or modified by the Scheduled Task wizard.
[WMI:ScheduledJobs]
disabled = 0
## Run once per day
interval = 86400
wql = SELECT Caption, Command, Description, InstallDate, InteractWithDesktop, JobId, JobStatus, Name, Notify, Priority, RunRepeatedly, Status FROM Win32_ScheduledJob
index = idx_appdev

## Services

## http://msdn.microsoft.com/en-us/library/aa394418(VS.85).aspx
## Lists all services registered on the system,if they are running,and the status
[WMI:Service]
disabled = 0
## Run once an hour
interval = 3600
wql = SELECT Name, Caption, State, Status, StartMode, StartName, PathName, Description FROM Win32_Service
index = idx_appdev


## Update
[WMI:InstalledUpdates]
disabled = 0
## Run once per day
interval = 86400
wql = SELECT Description, FixComments, HotFixID, InstalledBy, InstalledOn, ServicePackInEffect FROM Win32_QuickFixEngineering
index = idx_appdev


## Uptime
[WMI:Uptime]
disabled = 0
## Run once an hour
interval = 3600
wql = SELECT SystemUpTime FROM Win32_PerfFormattedData_PerfOS_System
index = idx_appdev

## index = idx_appdev


## Version
[WMI:Version]
disabled = 0
## Run once per day
interval = 86400
wql = SELECT Caption, ServicePackMajorVersion, ServicePackMinorVersion, Version FROM Win32_OperatingSystem
index = idx_appdev

And also the half-baked dashboard we use.

<form>
  <label>WMI Dashboard</label>
  <fieldset submitButton="true" autoRun="false">
    <input type="multiselect" token="hostname" searchWhenChanged="false">
      <label>Host Group</label>
      <search>
        <query>index=* sourcetype="WMI:Service" |dedup host |eval host=upper(host) |search host=* |sort -host</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <choice value="stwweb01 OR host=stwweb02 OR host=stwweb03 OR host=stwweb04">*STWWEB Prod</choice>
      <choice value="swf4d* host!=swf4d*q host!=swf4d*d">*4D Production</choice>
      <choice value="stwmt*">*STWMT</choice>
      <prefix>(</prefix>
      <suffix>)</suffix>
      <fieldForLabel>host</fieldForLabel>
      <fieldForValue>host</fieldForValue>
      <valuePrefix>host=</valuePrefix>
      <delimiter> OR </delimiter>
      <default>stwweb01 OR host=stwweb02 OR host=stwweb03 OR host=stwweb04</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <title>Summary Table</title>
        <searchString>index=* $hostname$  (sourcetype="WMI:InstalledUpdates" OR sourcetype="WMI:Uptime" OR sourcetype="WMI:Version")  | eventstats dc(HotFixID) as "Number of Patches" by host   | eval DaysUp=round(SystemUpTime/60/60/24,2)  |eventstats latest(DaysUp) as "Uptime" by host   | where sourcetype="WMI:Version"  |rex "Caption=(?&lt;OS&gt;.*)"  |stats latest(Uptime) as Uptime latest(OS) as OS latest(Version) as Version latest(ServicePackMajorVersion) as SP_Major# latest(ServicePackMinorVersion) as SP_Minor#  latest("Number of Patches") as "Number of Patches" by host |sort + host</searchString>
        <earliestTime>-36h</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">25</option>
      </table>
    </panel>
    <panel>
      <table>
        <title>Number of Patches (Sometimes breaks on summary table)</title>
        <searchString>$hostname$  (index=idx_appdev) sourcetype="WMI:InstalledUpdates" |stats dc(HotFixID) as "Number of Patches" by host</searchString>
        <earliestTime>-26h</earliestTime>
        <latestTime>now</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">false</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">column</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">30</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <title>OS Version</title>
        <searchString>index=* $hostname$ sourcetype="WMI:Version"   | dedup 1 host  | rex "Caption=(?&lt;OS&gt;.*)"  | eval OSVersion=OS."-SP ".ServicePackMajorVersion| top limit=20 OSVersion</searchString>
        <earliestTime>-24h@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">false</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">pie</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
      </chart>
    </panel>
    <panel>
      <chart>
        <title>7 Day Uptime Graph</title>
        <searchString>index=idx_appdev $hostname$ sourcetype="WMI:*" sourcetype="WMI:Uptime"  | eval DaysUp=round(SystemUpTime/60/60/24,2)| timechart span=1h avg(DaysUp) as Uptime by host useother=f limit=20</searchString>
        <earliestTime>-7d@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">false</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.nullValueMode">connect</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>Last WindowsUpdate (Work in progress)</title>
        <searchString>index=* $hostname$  sourcetype="WMI:InstalledUpdates" | eval epochtime=strptime(InstalledOn,"%m/%d/%Y")| eval Updated=strftime(epochtime,"20%y-%m-%d") |sort +Updated |stats max(Updated) as "Last Updates Installed" by host |sort + host</searchString>
        <earliestTime>-24h@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">20</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>Login Events</title>
        <search>
          <query>index=* $hostname$  index=idx_security sourcetype="WinEventLog:Security" Keywords="Audit Success"  (Account_Name=* NOT "ANONYMOUS LOGON" NOT svc* NOT *$ NOT - NOT IUSR_DATSTAT NOT SYSTEM NOT DefaultAppPool NOT webservice.external.weighttalkweb.com) (Security_ID=* NOT CHP\svc* NOT WEB\svc*) |eval LoginType=case(Logon_Type=3,"RPC",Logon_Type=4,"Batch",Logon_Type=5,"Service",Logon_Type=7,"Unlock",Logon_Type=10,"RDP/Terminal",Logon_Type=11,"Cached",Logon_Type=9,"New Credentials") |stats count(Account_Name) as "Login/Off Events" by Account_Name LoginType, host |sort + Account_Name</query>
          <earliest>-4h</earliest>
          <latest>now</latest>
        </search>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">10</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>KB installed by Host</title>
        <searchString>index=idx_appdev $hostname$ sourcetype="WMI:InstalledUpdates" | rex "Description=(?&lt;Type&gt;.*)" |stats dc(host) as "Hosts Installed On" by HotFixID Type |sort -"Hosts Installed On"</searchString>
        <earliestTime>-7d@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">20</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>By Update Type</title>
        <searchString>index=idx_appdev $hostname$ sourcetype="WMI:InstalledUpdates" | rex "Description=(?&lt;Update_Type&gt;.*)" |stats dc(HotFixID) as "Number" by Update_Type |sort - "Number"</searchString>
        <earliestTime>-24h@h</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">20</option>
      </table>
    </panel>
    <panel>
      <table>
        <title>Services Running</title>
        <searchString>index=idx_appdev $hostname$ sourcetype="WMI:Service" | rex "Caption=(?&lt;Name&gt;.*)"|stats dc(host) as "Hosts" by Name |sort + Hosts</searchString>
        <earliestTime>-4h@m</earliestTime>
        <latestTime>now</latestTime>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">20</option>
      </table>
    </panel>
  </row>
</form>

topic Re: Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)? in Splunk Search

Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)?

Re: Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)?

Re: Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)?

Re: Using perfmon and inputs.conf, how do we pull in data for specific processes (by name)?