How to find missing field values

[jdonn_splunk]

I want to use Splunk to tell me when a process is missing from a list of expected processes.

I have tried using eval and stats, but I can't figure out how to get it right. It is very easy to see when things are there, but hard to tell when they are not there.

[jdonn_splunk · Accepted solution]

This problem can be solved by leveraging a lookup table and a pretty simple search.

First, create a lookup file with your desired processes:

monitor_me.csv
sourcetype,process
proc,splunkd
proc,ntpd
proc,fsck_hfs
proc,backupd
proc,diskspaced
proc,webinspector
proc,Never Showing Up
proc,Splunk Can Monitor Things

| inputlookup monitor_me.csv 
| join type=outer process 
    [ search sourcetype= proc 
    | stats count by process] 
| table proc count 
| fillnull value=0 
| where count=0

The search results will look something like this, if the last two entries in the csv are not found:

process                 count
—————————————————————————
Never Showing Up             0
Splunk Can Monitor Things   0

Now, you can schedule this search and set up an alert.