Alerting

How to create an Alert - if the the value is greater than the threshold for next 10 mins, it should be alerted

smanojkumar
Path Finder
index=sap source=P* (EVENT_TYPE=abc)
| fields FDATE FTIME LDATE LTIME QDEEP QNAME FIRSTTID QSTATE EVENT_TYPE source
| eval earliestCT = strptime(strftime(now() + `utcdiff("America/Chicago")`,"00:00:00 %m/%d/%Y America/Chicago"),"%H:%M:%S %m/%d/%Y %Z"),
latestCT = strptime(strftime(now() + `utcdiff("America/Chicago")`,"23:59:59 %m/%d/%Y America/Chicago"),"%H:%M:%S %m/%d/%Y %Z"),
DateCT = strftime(now() + `utcdiff("America/Chicago")`,"%m/%d/%Y"),Created = strptime(FDATE." ".FTIME,"%Y%m%d %H%M%S"),
lastupdate=strptime(LDATE." ".LTIME,"%Y%m%d %H%M%S")
| where Created >= earliestCT AND Created <= latestCT
| dedup source EVENT_TYPE QNAME FIRSTTID
| stats sum(QDEEP) as TotalEntries values(DateCT) as DateCT by source EVENT_TYPE
| lookup Lookup_SAP_PERF_EntryThresholds.csv source EVENT_TYPE OUTPUTNEW Threshold LastAlertedDate
| where (tostring(DateCT) != tostring(LastAlertedDate)) AND match(Threshold,".+") AND (TotalEntries >= Threshold)




To add new requirement in the existing alert, When the entries are greater than threshold and staying for more than 10 mins and not reducing further then it should trigger.

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

That's not how alerts work.  We can't tell an alert to run for 10 minutes just in case certain events arrive.  Nor can an alert look into the future for events that have not yet arrived.

The usual practice in this use case is to have the alert look at the previous 10 minutes to see if the alert criteria was met during that time.  If so, the alert is triggered immediately.

---
If this reply helps you, Karma would be appreciated.
0 Karma

smanojkumar
Path Finder

Is that by using earliest and latest, By using the same query in sub search for earliest=10m latest=now, 
If there are any other methods, can you please say.(Query would help me)

0 Karma

richgalloway
SplunkTrust
SplunkTrust

There's no need for a subquery.  Run the a query over 10 minutes, using earliest=-10m.  Break the results into 1-minute chunks using bin.  Make your measurement and count how many are over the threshold.  If the count is 10 then trigger an alert.  See https://community.splunk.com/t5/All-Apps-and-Add-ons/Alert-if-Value-over-threshold-for-a-certain-per... for an example.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...