topic Re: Monitor specific set of Processes from WMI in Splunk Search

Monitor specific set of Processes from WMI

joshhenderson — Tue, 03 Jul 2012 06:36:02 GMT

Hi,

What I'm attempting to do is monitor a specific set of processes on a machine. For this, I am obtaining data from WMI (where wmi_type=LocalProcesses,) and checking how many instances of that a specific process has occurred within a specified time frame (which I choose depening on how often the data from that server gets updated.) If it is ever zero, then there is an error.

Below is a search query that I am using on a dashboard:

host=[HOST] wmi_type=LocalProcesses earliest=-5m Name="[PROCESS]" | stats count AS TimesDetected | rangemap field=TimesDetected severe=0-0 default=low

(Where [HOST] is the name of the particular machine I am checking for, and [PROCESS] is the name of the process. An example would be Server01 and CcmExec respectively.)

This works quite well, and I can add text only panels to my dashboard (within the search app) to monitor each process.

The problem with this is that, once more than a handful of processes are monitored, the dashboard get's very cluttered (since you can only have 2 panels per row, and each process takes up a panel.) I am looking for a way to display the data in a table, for only certain processes that I have specified.

In an attempt to do this, I have come up with the following query:

host=HTVMBI02 wmi_type=LocalProcesses earliest=-5m Name="System" OR "CcmExec" | stats count AS TimesDetected by Name

It works well when viewed as a table, and I can add as many processes as I want to monitor to the table. However, should the process not have any data, it will not be shown in the resulting table. I can understand why this is (there is no data for it,) but I am looking for a way to force it to say '0' for processes I've specified that it can't find data for. Is there a way to do this?

Thanks.

Re: Monitor specific set of Processes from WMI

dwaddle — Tue, 03 Jul 2012 19:24:39 GMT

One approach (an obviously not optimal one, but perhaps it will scale sufficiently) might be to use a lookup table and | append.

Suppose you set up a lookup table like this (call it processes.csv):

Name,TimesDetected
System,0
CcmExec,0

Now, you can make your search as follows:

host=HTVMBI02 wmi_type=LocalProcesses earliest=-5m 
[ | inputlookup processes.csv | fields Name ] 
| stats count AS TimesDetected by Name
| append [ | inputlookup processes.csv ]
| stats max(TimesDetected) as TimesDetected by Name

In theory, the lookup not only populates the base search but provides sentinel values to make it so you can be sure that every possible row has a "TimesDetected=0" value for display purposes.

Re: Monitor specific set of Processes from WMI

joshhenderson — Wed, 04 Jul 2012 04:31:52 GMT

Thanks, worked really well. I was on the right track (very close actually) as I was looking at lookup tables, but had issues uploading it (which I've since resolved) and I wasn't putting a 0 value in the second column. Thanks for the help.