topic Re: How to Set an Alert to Fire on The Total Count? in Splunk Search

How to Set an Alert to Fire on The Total Count?

skoelpin — Tue, 01 Sep 2015 18:54:42 GMT

I have an alert set which will compare the errors for the current day's previous hour to yesterday's previous hour..

So it will run a search every hour and count the number of errors from 1pm-2pm today and compare them to the number of errors received yesterday from 1pm-2pm. If the number of errors for today's hour is 25% higher than yesterday's hour of errors, then the alert will fire.

I set the search up and it's working as expected, but I believe I misconfigured the alert because I just got 2 alert emails. The first email is showing Calculate Tax errors, then the next alert email is showing the total number of errors. I want the alert to fire if ONLY if the total value for 'TodayLastHour' is 25% greater than 'YesterdayLastHour'.

Tax Call          |   TodayLastHour             | YesterdayLastHour
CalculateTax      |        290                  |      100
LookupTax         |        100                  |       90
TOTAL             |        390                  |      190

Here's my search:

index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | where TodayLastHour >  1.25 * YesterdayLastHour

Re: How to Set an Alert to Fire on The Total Count?

richgalloway — Tue, 01 Sep 2015 19:30:06 GMT

I would create a separate search that only reports the total error count and use that as the basis for the alert.

Re: How to Set an Alert to Fire on The Total Count?

skoelpin — Tue, 01 Sep 2015 20:10:43 GMT

I was considering this approach but I'm not sure if this method will return the expected results. I want the alert to show a table (The one I listed above) which will show how many errors in each tax call along with the total errors. But I only want the alert to fire if the total errors from 'TodayLastHour' is 25% more then 'YesterdayLastHour'

Re: How to Set an Alert to Fire on The Total Count?

somesoni2 — Tue, 01 Sep 2015 21:01:56 GMT

I guess you've configured your alert as per-result hence you get two email. If you just want one 1 alert and that too based on total value, then you would either have to update this current search (you'll loose granularity of TaxCall) OR create a new search. In any case following would be the query to use.

index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | table TodayLastHour YesterdayLastHour  | stats sum(*) as *  | where TodayLastHour >  1.25 * YesterdayLastHour

Re: How to Set an Alert to Fire on The Total Count?

jkat54 — Tue, 29 Sep 2020 07:12:54 GMT

I think @richgalloway is correct:

saved search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h
| stats count AS TodayLastHour
| append [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour]
| where TodayLastHour > 1.25 * YesterdayLastHour

trigger:
if results > 0

run a script: perlscript.that.calls.splunkd.and.triggers.report.email.search.pl

saved report / email search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | where TodayLastHour > 1.25 * YesterdayLastHour

trigger: always alert

So that the first one triggers a script that runs a saved search which triggers an email. The first one will only run when the where clause is met, and the 2nd search is ran by a perl script via splunkd api call.

Re: How to Set an Alert to Fire on The Total Count?

jkat54 — Tue, 29 Sep 2020 07:12:57 GMT

trigger:
if results > 0

run a script: perlscript.that.calls.splunkd.and.triggers.report.email.search

trigger: always alert

Re: How to Set an Alert to Fire on The Total Count?

woodcock — Wed, 02 Sep 2015 20:10:53 GMT

Go into your alert settings and change When triggered, execute actions to Once and it will give you your expected results.

Re: How to Set an Alert to Fire on The Total Count?

skoelpin — Wed, 02 Sep 2015 20:18:03 GMT

Worked as expected, thanks!!