How do I list machines that do not match my search?
"if" my script runs, a message is sent to splunk. The script runs once a week. I can easily see the details of my scripts, if it runs in splunk. So how do I list the machines that this script doesn't even start on? eg no entry sent to splunk.
This search does not list those who count is "zero", how do I list the "zero" machines?
"MyAppResults" | stats count by host | stats sum(count) as count by host
If I understood I should be using "inputlookup hosts.csv" but I'm not sure how to use it properly. I still cannot get it to list "zero" machines.
if you need to find the hosts in a monitoring perimeter that didin't send logs, you have to create a list of monitored hosts and use it for the control.
In few words:
| metasearch index=_internal | eval host=lower(host) | stats count BY host | append [ search | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ] | stats sum(count) AS total BY host | where count=0
With this search you check the hosts with Forwarder, if you want also check hosts without forwarder (e.g. network appliance) you have to use a different index,
if you want to filter your logs using some field, you cannot use "| metasearch".
You can run this search all the times you want, also very frequently (e.g. 5 minutes) to control that you continously have logs. without them you're blind!
About the lookup, you have two way to populate it:
The first solution is easier and requires less job, but in this way you also have less control on your monitoring perimeter than the second one.
Finding something that is not there is not Splunk's strong suit. See this blog entry for a good write-up on it.
Your lookup file will contain a list of all machines that run the script. Compare that list to the list of results you get from the "MyAppResults" search to find out which machines did not run.