Reporting

Trigger and send mail after 15th occurrence based on threshold breach

prsubramanian
New Member

Hi,
My requirement is : I have create real time schedule which will run for every 2 mins (cron schedule) based on the CPU Utilization crossed threshold value and mail should not trigger after 2 mins.
Mail has to trigger after 15th occurrence (ie after 15 * 2mins = 30 mins) if the threshold value crosses 98 %.

Please let me know how to check for the above condition to send mail after 15th occurrence if it cross threshold value.

Below is the query;
| table Host,"Process ID", "Instance"," % CPU Time" |sort " % CPU Time" desc | head 10
| eval "Alert Status" = case(('% CPU Time'<=manual_threshold3),"NORMAL", (' % CPU Time'>manual_threshold4),"CRITICAL", (' % CPU Time'>manual_threshold3 AND ' % CPU Time'<=manual_threshold4),"WARNING")
| search "Alert Status"="CRITICAL"
| table Host,"Process ID","Instance"," % CPU Time","Alert Status"

Thanks

0 Karma

woodcock
Esteemed Legend

Is this also you(r question)?
https://answers.splunk.com/answers/776314/how-to-send-alert-based-on-number-of-occurrences-b.html#an...

In any case, I have the same answer:
The way to do this is to write out the results of this search to a lookup file with ... | outputlookup. Then create another scheduled search that runs on a different cron schedule that mines the outputs of the first search using | inputlookup ... and applies your throttling/email logic. This second search (or even a third search) can trim/cleanup the lookup so that it doesn't grow out of control in size.

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...