Splunk Search
Highlighted

How do create an alert to trigger if my search returns a count that is greater than or equal to 20 per minute for 10 minutes?

Path Finder

Hi ,

We need to set up an alert to check if events with below format exists:

index=idx1 sourcetype=compass:services 503_SYS_ERR  Error

Condition:
If count is >= 20/minute for 10 min (If count is greater than 20 consistently for 10 min in a minute interval), then trigger an alert.

0 Karma
Highlighted

Re: How do create an alert to trigger if my search returns a count that is greater than or equal to 20 per minute for 10 minutes?

SplunkTrust
SplunkTrust

Try something like this. Below query will return result if the count per min (after timechart with span=1m) is 20 or more (| where count>=20) for 10 minutes (| where count=10)

index=idx1 sourcetype=compass:services 503_SYS_ERR  Error earliest=-10m@m latest=@m | timechart span=1m count | where count>=20 | stats count | where count=10

You can set this to run every minute.

View solution in original post

0 Karma
Highlighted

Re: How do create an alert to trigger if my search returns a count that is greater than or equal to 20 per minute for 10 minutes?

Path Finder

Hi,

If I use this query , it is not retrieving any results. May be because if we give |stats count it will only return us count and then condition |where count=10 which doesn't find any such values and henceis not retrieving any results.

Can you please suggest?

0 Karma
Highlighted

Re: How do create an alert to trigger if my search returns a count that is greater than or equal to 20 per minute for 10 minutes?

SplunkTrust
SplunkTrust

Try to remove add one portion of the search at a time and see if the results are what you want. If my understanding was correct, you want to alert if your condition was fulfilled (per min count >=20 for 10 continuous mins). This query will return results only when that condition is met.

0 Karma
Highlighted

Re: How do create an alert to trigger if my search returns a count that is greater than or equal to 20 per minute for 10 minutes?

Path Finder

Got it, yes condition is correct.

So I need to schedule this to run for every minute, where this would solve the purpose.

thanks.

0 Karma
Highlighted

Re: How do create an alert to trigger if my search returns a count that is greater than or equal to 20 per minute for 10 minutes?

Legend

If you want to run the query for a period greater than 10min, try this. Or @somesoni2 solution should work, if you schedule it to run every 10 mins

 index=idx1 sourcetype=compass:services 503_SYS_ERR  Error | timechart span=1m count as per_min_c |  streamstats count as c | eval grp=ceiling(c/10) | where per_min_c > 20 | eventstats reoccurrence by grp | where reoccurrence>=10
0 Karma