topic Re: Alerting for zero events in Alerting

Alerting for zero events

peter_gianusso — Mon, 28 Sep 2020 12:38:01 GMT

we want a search to execute every 20 minutes that looks at the past 20 minutes and ensures that events have been written to the log file by host.

This search returns no results.
sourcetype="PROD_PSI_HR_UNISYS_RTENGINE_DBG" | timechart count span=20m by host | where count=0

When I remove the "by host", the search below only works for a search for the past 4 hours but when I change it to 60 minutes, it stops returning results. It definitely doesn't work for the past 20 minutes.

sourcetype="PROD_PSI_HR_UNISYS_RTENGINE_DBG" | timechart count span=20m | where count=0

Any ideas?

Re: Alerting for zero events

yannK — Tue, 16 Oct 2012 15:54:08 GMT

If a host has no events, it will not be listed at all.

To proceed, you have to save a static list of the hosts, by example in a lookup file or a csv file. (See outputlookup, outputcsv, inputlookup, inputcsv)
and use this lookup as a | JOIN type=outer host to populate the other results, then the missing hosts will be displayed.
see http://docs.splunk.com/Documentation/Splunk/4.3.4/SearchReference/Join

here is an example with csv.
to generate the list once a week over all the week events :

earliest=-1w@w latest=@w | dedup host | stats count AS lastweek by host | outputcsv lastweekhostlist.csv

to use this list with last hour event list

FYI, we use the dedup host to keep only 1 event per hosts to be faster, but you could remove it and use the actual count of events.

Re: Alerting for zero events

peter_gianusso — Mon, 28 Sep 2020 12:38:04 GMT

sorry...lets forget the hosts problem...it's really the query sourcetype="PROD_PSI_HR_UNISYS_RTENGINE_DBG" | timechart count span=20m | where count=0 that's the problem

Re: Alerting for zero events

yannK — Tue, 16 Oct 2012 16:07:32 GMT

actually the timechart by host will return a table like :
timestamp host1 host2 host3
therefore there is not count columns to filter on.

try to replace by
| bucket _time span=20m | stats count by _time host
that will return a table like
timestamp count host