Splunk Search

Throttle Alerts for a table of results until end of the current day

njohnson7
Path Finder

I am trying to setup an alert which will run every hour and considers the data from the start of current day(earliest=@d) and send the summary of table results via email. As long as the first set of results are produced I would like to throttle it until end of the current day.

When I select the trigger option ONCE , i have the values like hours, days etc for which if select 24 or 1 respectively , I am afraid my alert will trigger again only after 24 hours or a day from the time it was triggered last.

When I evaluate a Date field and try to throttle it using the PER RESULT option and using Date as a suppressing field, the alert email only has the first line & the Grand Total line from my table summary - since my table of results are lost , I cant use this method.

Could you please advise a way to keep my table result structure while letting me to throttle until end of the current day.

Desired Result:

      SYSTEM      Qname        FAIL         TRYAGAIN  Grand Total
      sys1         ABC*        300            25        325
      sys1         DEF*        210            55        265
      sys1         GHI*        470            25        495
      sys1         JK*         250            25        275
               Grand Total     1230           130       1360

Query :

index=xyz sourcetype="abc" SYSTEM="sys1" 
| dedup client NAME STATUS 
| search STATUS=FAIL OR STATUS=TRYAGAIN 
| eval "Qname"=case(NAME like "ABC%" , "ABC*", NAME like "DEF%" ,"DEF*", NAME like "GHI%" , "GHI*" , NAME like "JK%","JK*", 1=1,"Others") 
| search "Qname"!=Others 
| eventstats count as Grandtotal 
| where Grandtotal >50 
| eval Date=strftime(_time, "%d/%m/%Y") 
| stats count(eval(STATUS="FAIL")) as "FAIL" , count(eval(STATUS="TRYAGAIN")) as "TRYAGAIN" values(Date) as Date by SYSTEM, "Qname" 
| table Date SYSTEM "Qname" FAIL TRYAGAIN
| addtotals fieldname="Grand Total" col=t row=t labelfield="Qname" label="Grand Total"
0 Karma

jacobpevans
Motivator

When I evaluate a Date field and try to throttle it using the PER RESULT option and using Date as a suppressing field, the alert email only has the first line & the Grand Total line from my table summary - since my table of results are lost , I cant use this method.

I'm not fully following this. Could you try doing that again but with _time to now() and values(Date) as Date by SYSTEM, "Qname" to by Date SYSTEM "Qname" e.g.:

 index=xyz sourcetype="abc" SYSTEM="sys1" 
 | dedup client NAME STATUS 
 | search STATUS=FAIL OR STATUS=TRYAGAIN 
 | eval "Qname"=case(NAME like "ABC%" , "ABC*", NAME like "DEF%" ,"DEF*", NAME like "GHI%" , "GHI*" , NAME like "JK%","JK*", 1=1,"Others") 
 | search "Qname"!=Others 
 | eventstats count as Grandtotal 
 | where Grandtotal >50 
 | eval _Date=strftime(now(), "%d/%m/%Y") 
 | stats count(eval(STATUS="FAIL")) as "FAIL" , count(eval(STATUS="TRYAGAIN")) as "TRYAGAIN" by _Date SYSTEM "Qname" 
 | table _Date SYSTEM "Qname" FAIL TRYAGAIN
 | addtotals fieldname="Grand Total" col=t row=t labelfield="Qname" label="Grand Total"
Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

njohnson7
Path Finder

@jacobevans thankyou but it is more or less similar to what I already tried. The results appears in the below format.

Date             SYSTEM      Qname        FAIL         TRYAGAIN  Grand Total
08/08/2019       sys1         ABC*        300            25        325
08/08/2019       sys1         DEF*        210            55        265
08/08/2019       sys1         GHI*        470            25        495
08/08/2019       sys1         JK*         250            25        275
            Grand Total     1230           130       1360

Now when I try to trigger the alerts using the "FOR EACH RESULT" option and giving Date as the field value for suppressing the alerts.

I only get the first line and the last line from my table of results since the date value is same:

 Date             SYSTEM      Qname        FAIL         TRYAGAIN  Grand Total
 08/08/2019       sys1         ABC*        300            25        325
                           Grand Total     1230           130       1360

I would like to send my entire table contents in the alerts in the format I have shown in the example , however I would like to suppress it only until end of the day and start over again as soon as the next day starts.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...