Alerting

Alert on missing events

cmisztur
Explorer

This is more of a comment than question. Based on some other QA this search works great to alert me when an event goes missing.

index="rigs" earliest=-25m latest=-11m 
| dedup rig 
| stats count AS last_period by rig 
| JOIN type=outer rig 
    [| search index="rigs" earliest=-10m latest=now 
    | dedup rig 
    | stats count as this_period by rig ] 
| fillnull value=0 this_period last_period 
| where last_period>0 AND this_period=0
Tags (1)
0 Karma

acharlieh
Influencer

But it seems rather inefficient because you're running two searches over the same data to get to two sides of a result that can be accomplished in only one search... For example:

index="rigs" earliest=-25m latest=now
| eval period=if(_time>relative_time(now(),"-10m"),"current","last")
| chart count over rig by period
| where last>0 AND current=0

And it gets indexers to categorize for you. (I might have my over and by terms mixed up on the chart command as I'm doing this by hand.)

If rig is an indexed field like host and you have no qualifiers that rely on search time field extractions, you could possibly get even better performance by using tstats and to skip rawdata expansion. (Again slight syntax issues possible here as by hand on iPhone, also not sure if doing prestats or just normal stats/chart on the results would be better.)

| tstats prestats=true count where index="rigs" earliest=-25m latest=now by host _time span=1m
| eval period=if(_time>relative_time(now(),"-10m"),"current","last") 
| chart count over host by period 
| where last>0 AND current=0

dgilmour22
New Member

acharlieh ..
Can you check your answer? When the events disappear, I'm finding that the "Current" column goes away, and so the subsequent where evaluation does not occur because current is null.. I tried to replace with the above "fillnull" command, but that didn't work for me. (although I'm a very newbie).

0 Karma

acharlieh
Influencer

You're correct that there is an assumption that at least one rig/host reported in the current period. As long as that is true, then current would be populated for all rigs.

If it is possible that no rigs reported in the current period (and there isn't a different alert for that...), then we can alter the where clause slightly and cover this case as well:

| where last>0 AND (isnull(current) OR current=0)

0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...