topic Re: alert if field missing over x mins in Alerting

alert if field missing over x mins

stephenmoorhous — Tue, 18 Apr 2017 09:57:55 GMT

Hi, for a given index and sourcetype I have a field called host which can be one of 4 values
There normally several log-lines for each host every minute
I would like an alert if one of the hosts stops logging for say 5 minutes

I have
index=myindex sourcetype=mysourcetype earliest=-5m | stats count by host

and the alert to trigger if the value is 0 - but if one of the hosts is no in the logs - you get
host count
host-a 128
host-b 143
host-d 122

and host-c is missing rather than 0 so nothing triggers 😞

any ideas other than 4 separate alerts?

Re: alert if field missing over x mins

gcusello — Tue, 18 Apr 2017 10:17:33 GMT

Hi stephenmoorhouse,
I suggest to you to insert your hosts list in a lookup (e.g. perimeter.csv) and then use this lookup for serching the missed ones, note that for only 4 host isn't mandatory to use a lookup, you could insert them in your search (host=host1 OR host=host2 OR host=host3 OR host=host4), but it's a best practice!
Every way try something like this:

index=myindex sourcetype=mysourcetype earliest=-5m 
| eval count=0, host=upper(host) 
| append [ | inputlookup perimeter.csv | eval host=upper(host) | stats count by host ]
| stats sum(count) AS Total by host
| where Total=0

In this way you have all hosts that didn't send logs in the last 5 minutes.

You can use it in an alert scheduled every 5 minutes or, without the last row, in a dashboard panel to show all monitored hosts, highlighting the missed ones.
You can also show this panel in a table with coloured icons.

Bye.
Giuseppe

Re: alert if field missing over x mins

Ravan — Tue, 18 Apr 2017 10:28:52 GMT

Can be done in multiple ways.

Host lookup (have a lookup uploaded with list of hosts)

|inputlookup lookup_hosts.csv|table host | join type=outer host [search index=myindex sourcetype=mysourcetype earliest=-5m | stats count by host ] | search NOT count =*

Increase the time frame.

index=myindex sourcetype=mysourcetype earliest=-15m|dedup host |table _time host |where _time  < now() - 300

Similar to above logic but consider using either _inernal licence metrics or tstats command.

Re: alert if field missing over x mins

neilfajardo15 — Wed, 06 Oct 2021 07:51:48 GMT

Hi @gcusello , im currently stuck with the same issue as this but the difference is I don't use input lookup for my query. I have seen this comment of yours and it is possible to add the host in the search as you say but im not sure how can i add it to this query because im also using the same but im not using host using other field from the event logs Thanks in advance

Re: alert if field missing over x mins

gcusello — Wed, 06 Oct 2021 08:03:13 GMT

Hi @neilfajardo15,

if you have few hosts is possible to insert them in the search, even if I always hint to use the lookup to have a more maintenable object.

so, if you have three hosts, you could run something like this:

Ciao.

Giuseppe

P.S.: next time, please, open a new question.