Splunk Search

How to Set an Alert to Fire on The Total Count?

SplunkTrust
SplunkTrust

I have an alert set which will compare the errors for the current day's previous hour to yesterday's previous hour..

So it will run a search every hour and count the number of errors from 1pm-2pm today and compare them to the number of errors received yesterday from 1pm-2pm. If the number of errors for today's hour is 25% higher than yesterday's hour of errors, then the alert will fire.

I set the search up and it's working as expected, but I believe I misconfigured the alert because I just got 2 alert emails. The first email is showing Calculate Tax errors, then the next alert email is showing the total number of errors. I want the alert to fire if ONLY if the total value for 'TodayLastHour' is 25% greater than 'YesterdayLastHour'.

Tax Call          |   TodayLastHour             | YesterdayLastHour
CalculateTax      |        290                  |      100
LookupTax         |        100                  |       90
TOTAL             |        390                  |      190

Here's my search:

index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | where TodayLastHour >  1.25 * YesterdayLastHour
1 Solution

Esteemed Legend

Go into your alert settings and change When triggered, execute actions to Once and it will give you your expected results.

View solution in original post

Esteemed Legend

Go into your alert settings and change When triggered, execute actions to Once and it will give you your expected results.

View solution in original post

SplunkTrust
SplunkTrust

Worked as expected, thanks!!

0 Karma

SplunkTrust
SplunkTrust

saved search:
index=vertex7-access RTGError="500" earliest=-1h@h latest=@h
| stats count AS TodayLastHour
| append [search index=vertex7-access RTG
Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour]
| where TodayLastHour > 1.25 * YesterdayLastHour

trigger:
if results > 0

run a script: perlscript.that.calls.splunkd.and.triggers.report.email.search

saved report / email search:
index=vertex7-access RTGError="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTGTax | addtotals col=t | table RTGTax, TodayLastHour | appendcols [search index=vertex7-access RTGError="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTGTax | addtotals col=t | table RTGTax, TodayLastHour, YesterdayLastHour] | where TodayLastHour > 1.25 * YesterdayLastHour

trigger: always alert

So that the first one triggers a script that runs a saved search which triggers an email. The first one will only run when the where clause is met, and the 2nd search is ran by a perl script via splunkd api call.

SplunkTrust
SplunkTrust

I guess you've configured your alert as per-result hence you get two email. If you just want one 1 alert and that too based on total value, then you would either have to update this current search (you'll loose granularity of TaxCall) OR create a new search. In any case following would be the query to use.

index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | table TodayLastHour YesterdayLastHour  | stats sum(*) as *  | where TodayLastHour >  1.25 * YesterdayLastHour
0 Karma

SplunkTrust
SplunkTrust

I would create a separate search that only reports the total error count and use that as the basis for the alert.

---
If this reply helps you, an upvote would be appreciated.

SplunkTrust
SplunkTrust

I was considering this approach but I'm not sure if this method will return the expected results. I want the alert to show a table (The one I listed above) which will show how many errors in each tax call along with the total errors. But I only want the alert to fire if the total errors from 'TodayLastHour' is 25% more then 'YesterdayLastHour'

0 Karma

SplunkTrust
SplunkTrust

I think @richgalloway is correct:

saved search:
index=vertex7-access RTGError="500" earliest=-1h@h latest=@h
| stats count AS TodayLastHour
| append [search index=vertex7-access RTG
Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour]
| where TodayLastHour > 1.25 * YesterdayLastHour

trigger:
if results > 0

run a script: perlscript.that.calls.splunkd.and.triggers.report.email.search.pl

saved report / email search:
index=vertex7-access RTGError="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTGTax | addtotals col=t | table RTGTax, TodayLastHour | appendcols [search index=vertex7-access RTGError="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTGTax | addtotals col=t | table RTGTax, TodayLastHour, YesterdayLastHour] | where TodayLastHour > 1.25 * YesterdayLastHour

trigger: always alert

So that the first one triggers a script that runs a saved search which triggers an email. The first one will only run when the where clause is met, and the 2nd search is ran by a perl script via splunkd api call.

0 Karma