Alert on missing events

[cmisztur]

This is more of a comment than question. Based on some other QA this search works great to alert me when an event goes missing.

index="rigs" earliest=-25m latest=-11m 
| dedup rig 
| stats count AS last_period by rig 
| JOIN type=outer rig 
    [| search index="rigs" earliest=-10m latest=now 
    | dedup rig 
    | stats count as this_period by rig ] 
| fillnull value=0 this_period last_period 
| where last_period>0 AND this_period=0

[acharlieh]

But it seems rather inefficient because you're running two searches over the same data to get to two sides of a result that can be accomplished in only one search... For example:

index="rigs" earliest=-25m latest=now
| eval period=if(_time>relative_time(now(),"-10m"),"current","last")
| chart count over rig by period
| where last>0 AND current=0

And it gets indexers to categorize for you. (I might have my over and by terms mixed up on the chart command as I'm doing this by hand.)

If rig is an indexed field like host and you have no qualifiers that rely on search time field extractions, you could possibly get even better performance by using tstats and to skip rawdata expansion. (Again slight syntax issues possible here as by hand on iPhone, also not sure if doing prestats or just normal stats/chart on the results would be better.)

| tstats prestats=true count where index="rigs" earliest=-25m latest=now by host _time span=1m
| eval period=if(_time>relative_time(now(),"-10m"),"current","last") 
| chart count over host by period 
| where last>0 AND current=0

[dgilmour22]

acharlieh ..
Can you check your answer? When the events disappear, I'm finding that the "Current" column goes away, and so the subsequent where evaluation does not occur because current is null.. I tried to replace with the above "fillnull" command, but that didn't work for me. (although I'm a very newbie).

[acharlieh]

You're correct that there is an assumption that at least one rig/host reported in the current period. As long as that is true, then current would be populated for all rigs.

If it is possible that no rigs reported in the current period (and there isn't a different alert for that...), then we can alter the where clause slightly and cover this case as well:

| where last>0 AND (isnull(current) OR current=0)