Reporting

Trigger and send mail after 15th occurrence based on threshold breach

prsubramanian
New Member

Hi,
My requirement is : I have create real time schedule which will run for every 2 mins (cron schedule) based on the CPU Utilization crossed threshold value and mail should not trigger after 2 mins.
Mail has to trigger after 15th occurrence (ie after 15 * 2mins = 30 mins) if the threshold value crosses 98 %.

Please let me know how to check for the above condition to send mail after 15th occurrence if it cross threshold value.

Below is the query;
| table Host,"Process ID", "Instance"," % CPU Time" |sort " % CPU Time" desc | head 10
| eval "Alert Status" = case(('% CPU Time'<=manual_threshold3),"NORMAL", (' % CPU Time'>manual_threshold4),"CRITICAL", (' % CPU Time'>manual_threshold3 AND ' % CPU Time'<=manual_threshold4),"WARNING")
| search "Alert Status"="CRITICAL"
| table Host,"Process ID","Instance"," % CPU Time","Alert Status"

Thanks

0 Karma

woodcock
Esteemed Legend

Is this also you(r question)?
https://answers.splunk.com/answers/776314/how-to-send-alert-based-on-number-of-occurrences-b.html#an...

In any case, I have the same answer:
The way to do this is to write out the results of this search to a lookup file with ... | outputlookup. Then create another scheduled search that runs on a different cron schedule that mines the outputs of the first search using | inputlookup ... and applies your throttling/email logic. This second search (or even a third search) can trim/cleanup the lookup so that it doesn't grow out of control in size.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...