Alerting

How to configure real-time per-event alerts that trigger once on the same event ?

dhtran
Loves-to-Learn Lots

Hello,

I'm trying to figure out how to use Splunk to monitor payments processing, one of the business rules is to trigger 1 alert (and only 1) per payment as soon as it is "late".
a late payment means it is not processed in a predefined time window.

I have the search query that returns the results I needed.

But the challenges/prerequisites are :
- there's no per-event alert in Splunk, only per-result, which means a search query that returns 2 events will trigger 1 alert.
- having a search query that returns only 1 late payment at a time, in my case, is not possible.
- plus, I have a KPI "Nb of late payments" that needs to be decreased if the alerts on payments are deleted (via "Delete" action in Triggered Alert page).

Ex of a scenario :
I have 10 ongoing late payments, i want to yield 10 alerts individually. Then, if I delete 1 alert, I need to somehow "acknowledge" the payment to tell Splunk to :
1) stop yielding alert on this payment
2) add some data/flag/boolean to the payment so I can use it to filter the KPI to decrease its value (ex : search alert_acked=false")

Is it possible in Splunk to handle easily this scenario ?
Is there another way to achieve the same functionality ?

Thanks in advance for your help.

Labels (1)
0 Karma

to4kawa
Ultra Champion

make dashboard, output your confirm to csv, and make the query that check csv , search and fire alert.

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...