How do create an alert to trigger if my search returns a count that is greater than or equal to 20 per minute for 10 minutes?

[splunker9999]

Hi ,

We need to set up an alert to check if events with below format exists:

index=idx1 sourcetype=compass:services 503_SYS_ERR  Error

Condition:
If count is >= 20/minute for 10 min (If count is greater than 20 consistently for 10 min in a minute interval), then trigger an alert.

[sundareshr]

If you want to run the query for a period greater than 10min, try this. Or @somesoni2 solution should work, if you schedule it to run every 10 mins

 index=idx1 sourcetype=compass:services 503_SYS_ERR  Error | timechart span=1m count as per_min_c |  streamstats count as c | eval grp=ceiling(c/10) | where per_min_c > 20 | eventstats reoccurrence by grp | where reoccurrence>=10

[somesoni2 · Accepted solution]

Try something like this. Below query will return result if the count per min (after timechart with span=1m) is 20 or more (| where count>=20) for 10 minutes (| where count=10)

index=idx1 sourcetype=compass:services 503_SYS_ERR  Error earliest=-10m@m latest=@m | timechart span=1m count | where count>=20 | stats count | where count=10

You can set this to run every minute.

[splunker9999]

Hi,

If I use this query , it is not retrieving any results. May be because if we give |stats count it will only return us count and then condition |where count=10 which doesn't find any such values and henceis not retrieving any results.

Can you please suggest?

[somesoni2]

Try to remove add one portion of the search at a time and see if the results are what you want. If my understanding was correct, you want to alert if your condition was fulfilled (per min count >=20 for 10 continuous mins). This query will return results only when that condition is met.

[splunker9999]

Got it, yes condition is correct.

So I need to schedule this to run for every minute, where this would solve the purpose.

thanks.