Alert when total number of events are above 20 per second, continuously for 5 mins

[VS0909]

I have to trigger an alert if total number of events are above 20 per second, continuously for 5 mins.

Query :

index=abc  host = "123" earliest= -5m latest=now |  bucket _time span=1s | stats count  as Total by _time 

Alert should trigger only when 'Total' is above 20 for all the query results.

[VS0909]

Appreciate your reply!

@ITWhisperer  There could be a scenario when there is no data between five minutes,  in that scenario it will trigger a false alert as there will be zero results. Can you please help to fix this scenario.

Thanks!

[VS0909]

Appreciate your reply!

@ITWhisperer  There could be a scenario when there is no data between five minutes,  in that scenario it will trigger a false alert as there will be zero results. Can you please help to fix this scenario.

Thanks!

[VS0909]

Thanks for the reply!

I have one query @ITWhisperer . If we consider time from 1:00:00 to 1:05:00 and there are no events between 1:03 to 1:04, then there will be no data for this time frame in the query results.  In that case alert may trigger even when condition is not met. Is there any way to check that event total is displayed for every second, and if data is not present for a time duration, it should take total for that time duration as 0.

Please advise.

[ITWhisperer]

You could try

index=abc  host = "123" earliest= -5m latest=now 
|  bucket _time span=1s 
| stats count  as Total by _time
| makecontinuous _time
| eval Total=coalesce(Total,0)
| where Total <= 20
| appendpipe [stats count | where count = 0]
| eval allover20 = if(count = 0,"True",null)
| fields allover20

[VS0909]

Thanks for the reply, but I want the total count of events in every second for 5 mins, then trigger an  alert when 'Total' is above 20 for all the query results. Can you please help with that.

[ITWhisperer]

index=abc  host = "123" earliest= -5m latest=now 
|  bucket _time span=1s 
| stats count  as Total by _time
| makecontinuous _time span=1s
| eval Total=coalesce(Total,0)
| appendpipe [stats count | where count = 0 | rename count as Total]
| where Total <= 20
| appendpipe [stats count | where count = 0]
| eval allover20 = if(count = 0,"True",null)
| fields allover20

[ITWhisperer]

index=abc  host = "123" earliest= -5m latest=now 
|  bucket _time span=1s 
| stats count  as Total by _time
| where Total <= 20
| appendpipe [stats count | where count = 0]
| eval allover20 = if(count = 0,"True",null)
| fields allover20

[VS0909]

Thanks for the reply! Can you please help me understand that this query will help trigger an alert only when count is above 20 for all the rows returned. I am confused why are we doing " where count=0" ? Please advise

[ITWhisperer]

I flipped the query on its head, given that you want all counts to be over 20, if any are 20 or less, then not all are over 20, so if any rows remain you don't want to alert, it there are no rows (with count 20 or less), you want a result, which is what the appendpipe does. Then you simply eval a field and that is your trigger for the alert

[VS0909]

Appreciate your reply!

I understood that part, but why are we doing  "where count=0 "

in appendpipe command ?

Thanks again!

[ITWhisperer]

| where Total <= 20

will remove all the events if they are all over 20, so you could try alerting for zero results, alternatively, the appendpipe creates an event when there are no events, so you could alert on that.

[VS0909]

Appreciate your reply!

I understood that part, but why are we doing  "where count=0 "

in appendpipe command ? @ITWhisperer 

Thanks again!

[isoutamo]

You could try streamstats with window parameter.
https://docs.splunk.com/Documentation/Splunk/8.2.0/SearchReference/Streamstats