Alerting

How to create an Alert - if the the value is greater than the threshold for next 10 mins, it should be alerted

smanojkumar
Communicator
index=sap source=P* (EVENT_TYPE=abc)
| fields FDATE FTIME LDATE LTIME QDEEP QNAME FIRSTTID QSTATE EVENT_TYPE source
| eval earliestCT = strptime(strftime(now() + `utcdiff("America/Chicago")`,"00:00:00 %m/%d/%Y America/Chicago"),"%H:%M:%S %m/%d/%Y %Z"),
latestCT = strptime(strftime(now() + `utcdiff("America/Chicago")`,"23:59:59 %m/%d/%Y America/Chicago"),"%H:%M:%S %m/%d/%Y %Z"),
DateCT = strftime(now() + `utcdiff("America/Chicago")`,"%m/%d/%Y"),Created = strptime(FDATE." ".FTIME,"%Y%m%d %H%M%S"),
lastupdate=strptime(LDATE." ".LTIME,"%Y%m%d %H%M%S")
| where Created >= earliestCT AND Created <= latestCT
| dedup source EVENT_TYPE QNAME FIRSTTID
| stats sum(QDEEP) as TotalEntries values(DateCT) as DateCT by source EVENT_TYPE
| lookup Lookup_SAP_PERF_EntryThresholds.csv source EVENT_TYPE OUTPUTNEW Threshold LastAlertedDate
| where (tostring(DateCT) != tostring(LastAlertedDate)) AND match(Threshold,".+") AND (TotalEntries >= Threshold)




To add new requirement in the existing alert, When the entries are greater than threshold and staying for more than 10 mins and not reducing further then it should trigger.

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

That's not how alerts work.  We can't tell an alert to run for 10 minutes just in case certain events arrive.  Nor can an alert look into the future for events that have not yet arrived.

The usual practice in this use case is to have the alert look at the previous 10 minutes to see if the alert criteria was met during that time.  If so, the alert is triggered immediately.

---
If this reply helps you, Karma would be appreciated.
0 Karma

smanojkumar
Communicator

Is that by using earliest and latest, By using the same query in sub search for earliest=10m latest=now, 
If there are any other methods, can you please say.(Query would help me)

0 Karma

richgalloway
SplunkTrust
SplunkTrust

There's no need for a subquery.  Run the a query over 10 minutes, using earliest=-10m.  Break the results into 1-minute chunks using bin.  Make your measurement and count how many are over the threshold.  If the count is 10 then trigger an alert.  See https://community.splunk.com/t5/All-Apps-and-Add-ons/Alert-if-Value-over-threshold-for-a-certain-per... for an example.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...