Hi Splunkers, I am working on an alert which calculates the error rate (> 30%)and send the alerts to pagerduty via API index="test1" source="mylogs" NOT "TEST" earliest=-30m latest=now (":search-result match-indicator=\"PASS\" *******query******* | stats count(id) AS OKStats | appendcols [ search source=mylogs (":search-result match-indicator=\"ERROR\" *******query*******) NOT "TEST" earliest=-30m latest=now | stats count(id) AS ERRORStats] | eval TotalTransactions = OKStats + ERRORStats | eval ErrorRate = if(TotalTransactions > 0, round((DFAT_AP_ERR / TotalTransactions) * 100, 2), 0) | where ErrorRate >= 30 | eval dedup_key="HighErrorRate" | table ErrorRate, dedup_key Now to clear the alert, I created another alert(<30%) index="serverlogs" source="mylogs" NOT "TEST" earliest=-30m latest=now (":search-result match-indicator=\"PASS\" ************myqueryparams************ | stats count(id) AS OKStats | appendcols [ search source=mylogs (":search-result match-indicator=\"ERROR\" ************myqueryparams************) NOT "TEST" earliest=-30m latest=now | stats count(id) AS ERRORStats] | bin _time span=5m | eval ErrorPercent = if((OKStats + ERRORStats) > 0, round(ERRORStats / (OKStats + ERRORStats) * 100, 2), 0) | sort -_time | streamstats window=2 latest(ErrorPercent) as latest_percent, latest(_time) as latest_time, earliest(ErrorPercent) as earliest_percent | where latest_time = _time AND latest_percent < 30 AND earliest_percent >= 30 | head 1 | eval dedup_key="HighErrorRate" | table latest_percent, earliest_percent, dedup_key I created two conditions and sending to pagerduty(number of rows >1),running every 30min and enabled throttle. I do not see the second alert working or clearing the alert. Any advice how to achieve the clearing of alerts which means the alert should be cleared on pagerduty. Currently creating python script is out of scope due to security reasons. Hence, was trying it via splunk query. Regards, Amit
... View more