We need to set up an alert to check if events with below format exists:
index=idx1 sourcetype=compass:services 503_SYS_ERR Error
If count is >= 20/minute for 10 min (If count is greater than 20 consistently for 10 min in a minute interval), then trigger an alert.
Try something like this. Below query will return result if the count per min (after timechart with span=1m) is 20 or more (| where count>=20) for 10 minutes (| where count=10)
index=idx1 sourcetype=compass:services 503_SYS_ERR Error earliest=-10m@m latest=@m | timechart span=1m count | where count>=20 | stats count | where count=10
You can set this to run every minute.
If I use this query , it is not retrieving any results. May be because if we give |stats count it will only return us count and then condition |where count=10 which doesn't find any such values and henceis not retrieving any results.
Can you please suggest?
Try to remove add one portion of the search at a time and see if the results are what you want. If my understanding was correct, you want to alert if your condition was fulfilled (per min count >=20 for 10 continuous mins). This query will return results only when that condition is met.
Got it, yes condition is correct.
So I need to schedule this to run for every minute, where this would solve the purpose.
If you want to run the query for a period greater than 10min, try this. Or @somesoni2 solution should work, if you schedule it to run every 10 mins
index=idx1 sourcetype=compass:services 503_SYS_ERR Error | timechart span=1m count as per_min_c | streamstats count as c | eval grp=ceiling(c/10) | where per_min_c > 20 | eventstats reoccurrence by grp | where reoccurrence>=10