topic Re: Create an alert if error rate is over 10 ( or X number ) for 15min continusly. in Splunk Search

Create an alert if error rate is over 10 ( or X number ) for 15min continusly.

dpdwibedy — Thu, 13 Aug 2020 12:25:08 GMT

Hi There,

I have this query that reports the status code error rates.

index=apache_core userAgent!="nginx/*" source="*access.log*" requestURI!="/web/app*" NOT (requestURI="/api/xyz/*" OR requestURI="/api/yyy/*" AND statusCode=404) earliest=-30m latest=now | stats count(eval(statusCode>=400)) as errors, count as total | eval ErrorRate = errors * 100 / total | fields ErrorRate

This works , but looks like it is taking an average of error rate in last 30 mins and reports over the threshold if there is a onetime spike...

What I want is to Alert , only if the Error Rate is higher than threshold for continuous 10mins in last 30mins OR 1 hr.

How can I achieve that ?

Thanks,

Re: Create an alert if error rate is over 10 ( or X number ) for 15min continusly.

impurush — Thu, 13 Aug 2020 14:47:06 GMT

@dpdwibedy
This will not solve the exact solution which you are looking but we are handling the same kind of situation in our environment as below

index=apache_core userAgent!="nginx/*" source="*access.log*" requestURI!="/web/app*" NOT (requestURI="/api/xyz/*" OR requestURI="/api/yyy/*" AND statusCode=404) earliest=-30m latest=now | timechart span=5m count(eval(statusCode>=400)) as errors, count as total | eval ErrorRate = errors * 100 / total | fields _time,ErrorRate | where ErrorRate > Threshold

And if you want to send an alert if it is greater than 15 mins, then you need to configure the alert as trigger the alert if the result has more than 2 rows. Then the alert will trigger if it 3 times crosses the threshold in the 30 mins window.

Explanation: The above query split the 30 minutes window into 5 mins window and we are checking if 3 (5 mins) window is greater than the threshold, the alert will trigger. So it solves our problem.

Re: Create an alert if error rate is over 10 ( or X number ) for 15min continusly.

isoutamo — Thu, 13 Aug 2020 18:33:19 GMT

have you tried streamstats with time_window?

r. Ismo

Re: Create an alert if error rate is over 10 ( or X number ) for 15min continusly.

dpdwibedy — Thu, 13 Aug 2020 21:33:13 GMT

@isoutamo : Nope.

I am not good in Splunk queries . Could you give me an example ?

Thanks,

Re: Create an alert if error rate is over 10 ( or X number ) for 15min continusly.

dpdwibedy — Tue, 25 Aug 2020 17:47:25 GMT

Hi All,

I am still facing issue here . Can some one give me some examples ?

Thanks,

Re: Create an alert if error rate is over 10 ( or X number ) for 15min continusly.

thambisetty — Tue, 25 Aug 2020 19:06:42 GMT

index=apache_core userAgent!="nginx/*" source="*access.log*" requestURI!="/web/app*" NOT (requestURI="/api/xyz/*" OR requestURI="/api/yyy/*" AND statusCode=404) earliest=-30m latest=now

| bin _time span=10m

| stats count(eval(statusCode>=400)) as errors, count as total by _time

| eval ErrorRate = errors * 100 / total | fields ErrorRate _time

| where ErrorRate > yourthreshhold

Re: Create an alert if error rate is over 10 ( or X number ) for 15min continusly.

isoutamo — Tue, 25 Aug 2020 20:51:51 GMT

It should be something like this. I haven't your data so I cannot test exactly with it.

index=apache_core userAgent!="nginx/*" source="*access.log*" requestURI!="/web/app*" NOT (requestURI="/api/xyz/*" OR requestURI="/api/yyy/*" AND statusCode=404) earliest=-30m latest=now | streamstats time_span=10m count(eval(statusCode>=400)) as errors, count as total | eval ErrorRate = errors * 100 / total | fields _time,ErrorRate | where ErrorRate > Threshold

r. Ismo