I'm trying to figure out how to use Splunk to monitor payments processing, one of the business rules is to trigger 1 alert (and only 1) per payment as soon as it is "late". a late payment means it is not processed in a predefined time window.
I have the search query that returns the results I needed.
But the challenges/prerequisites are : - there's no per-event alert in Splunk, only per-result, which means a search query that returns 2 events will trigger 1 alert. - having a search query that returns only 1 late payment at a time, in my case, is not possible. - plus, I have a KPI "Nb of late payments" that needs to be decreased if the alerts on payments are deleted (via "Delete" action in Triggered Alert page).
Ex of a scenario : I have 10 ongoing late payments, i want to yield 10 alerts individually. Then, if I delete 1 alert, I need to somehow "acknowledge" the payment to tell Splunk to : 1) stop yielding alert on this payment 2) add some data/flag/boolean to the payment so I can use it to filter the KPI to decrease its value (ex : search alert_acked=false")
Is it possible in Splunk to handle easily this scenario ? Is there another way to achieve the same functionality ?