Scheduling conditional alert

[parthiban]

Hi team,

I have the following search code, and I want to trigger an alert when the condition is 'OFFLINE'. Note that we receive logs every 2 minutes, and the alert should be triggered only once; subsequent alerts should be suppressed. Similarly, when the condition becomes 'ONLINE', I want to trigger an alert only once, with subsequent alerts being suppressed. I hope my requirement is clear.

index= "XXXX" invoked_component="YYYYY" "Genesys system is available"
| spath input=_raw output=new_field path=response_details.response_payload.entities{}
| mvexpand new_field
| fields new_field
| spath input=new_field output=serialNumber path=serialNumber
| spath input=new_field output=onlineStatus path=onlineStatus
| where serialNumber!=""
| lookup Genesys_Monitoring.csv serialNumber
| where Country="Egypt"
| stats
count(eval(onlineStatus="OFFLINE")) AS offline_count
count(eval(onlineStatus="ONLINE")) AS online_count

| fillnull value=0 offline_count
| fillnull value=0 online_count
| eval condition=case(
offline_count=0 AND online_count>0,"ONLINE",
offline_count>0 AND online_count=0,"OFFLINE",
offline_count>0 AND online_count>0 AND online_count>offline_count, "OFFLINE",
offline_count>0 AND online_count>0 AND offline_count>online_count, "OFFLINE",
offline_count=0 AND online_count=0, "No data")
| search condition="OFFLINE" OR condition="ONLINE"
| table condition

[dtburrows3 · Accepted solution]

You may be able to use a streamstats method for this instead if you don't want to deal with lookups.

Although, you may need to set up alert throttling depending on how big your search time window is.

| search index="XXXX" invoked_component="YYYYY" "Genesys system is available"
    | spath input=_raw output=new_field path=response_details.response_payload.entities{}
    | mvexpand new_field
    | fields new_field
    | spath input=new_field output=serialNumber path=serialNumber
    | spath input=new_field output=onlineStatus path=onlineStatus
    | where serialNumber!=""
    | lookup Genesys_Monitoring.csv serialNumber
    | where Country="Egypt"
    | sort 0 +Country, +serialNumber, +_time
    | streamstats window=2
        first(onlineStatus) as previous_onlineStatus,
        last(onlineStatus) as next_onlineStatus
            by serialNumber  
    | table _time, Country, serialNumber, onlineStatus, previous_onlineStatus, next_onlineStatus
    | eval
        trigger_condition=if(
            NOT 'next_onlineStatus'=='previous_onlineStatus',
                1,
                0
            ),
        scenario=case(
            'previous_onlineStatus'=="ONLINE" AND 'next_onlineStatus'=="OFFLINE", "Host's online status went down",
            'previous_onlineStatus'=="OFFLINE" AND 'next_onlineStatus'=="ONLINE", "Hosts's online status came up"
            )
    ``` Set up alert throttle on Country, serialNumber & _time ```
    | where 'trigger_condition'>0

[parthiban]

Hi @dtburrows3 

Thanks a lot for your support, it is working as expected. 1000 Karma point to you.

[dtburrows3]

I think utilizing a lookup to track timestamps and changes of the onlineStatus by serialNumber would work for this.

| search index="XXXX" invoked_component="YYYYY" "Genesys system is available"
    | spath input=_raw output=new_field path=response_details.response_payload.entities{}
    | mvexpand new_field
    | fields new_field
    | spath input=new_field output=serialNumber path=serialNumber
    | spath input=new_field output=onlineStatus path=onlineStatus
    | where serialNumber!=""
    | lookup Genesys_Monitoring.csv serialNumber
    | where Country="Egypt"
    | stats
        latest(onlineStatus) as latestOnlineStatus,
        latest(_time) as latestStatusEpoch
            by serialNumber, Country
    | lookup host_online_status_tracking serialNumber, Country OUTPUT latestOnlineStatus as previousLatestOnlineStatus, latestStatusEpoch as previousLatestStatusEpoch
    | inputlookup append=true host_online_status_tracking
    | stats
        first(latest*) as latest*,
        first(previous*) as previous*
            by serialNumber, Country
    | outputlookup host_online_status_tracking
    | where ('latestStatusEpoch'>'previousLatestStatusEpoch' OR isnull(previousLatestStatusEpoch)) AND NOT 'latestOnlineStatus'=='previousLatestOnlineStatus'

 
The lookup is referenced to pull in the latest onlineStatus and timestamp from the previous time this search was run. You can see that there is an outputlookup that updates the lookup everytime with the most up-to-date data about each unique serialNumber. 

The final where clause is what will determine if the alert will fire or not while the lookup itself should stay updated with onlineStatuses. The idea is that the alert should only fire once after the onlineStatus changes since the lookup will be updated as well.

Run 1:
    Serial_1: Status=Up
        Exported to lookup ----> Serial_1: Status=Up
Run 2:
    Serial_1: Status=Down <----> Previous_From_Lookup=Up
        Alert Fires
        Exported to lookup ----> Serial_1: Status=Down
Run 3:
    Serial_1: Status=Down <----> Previous_From_Lookup=Down
        Alert doesn't fire since statuses match
        Exported to lookup ----> Serial_1: Status=Down
Run 4:
    Serial_1: Status=Up <----> Previous_From_Lookup=Down
        Alert Fires
        Exported to lookup ----> Serial_1=Up
.
.
.
        
That is the idea anyways. Hope this helps!

[parthiban]

Hi @richgalloway 

Can you please help me on this above requirement?