topic Re: realtime alert for each event when reaching x number in y mins in Splunk Search

realtime alert for each event when reaching x number in y mins

jdinze — Thu, 22 Feb 2018 09:26:51 GMT

I am trying to configure a real time alert that will fire off one alert for each event found in a search. I want one alert per event, which i think i can do. the catch is i only want this to happen when there are 10 or more events in a specified time window (like 10 or more events in 5 mins).

I tried setting up a realtime alert with the following parameters, but it seems like the results aren't consistent. am i doing this completely wrong?

(basically just searching an index for alerts, this index shouldn't have many but i want to know when there are 10 or more events in 5 mins and what each one is)
Search: index=test
Trigger Condition: Number of results > 10, in 5 min, trigger for each result

This requires a throttle, but i dont want one so i set the field to one that wouldnt exist and the smallest suppression timer.

Throttle: suppression field = "none"
suppress triggering for 1 sec

Thanks,
splunk noob

Re: realtime alert for each event when reaching x number in y mins

mayurr98 — Thu, 22 Feb 2018 10:17:45 GMT

So you can modify your search as

index=test  | table _raw

What trigger actions you are using?
If you are using email then you need to attach csv/pdf in order to see raw events
If you want to see on Splunk then you need to choose Add to triggered alerts as alert action

Let me know if this helps!

Re: realtime alert for each event when reaching x number in y mins

jdinze — Thu, 22 Feb 2018 14:30:17 GMT

I want to try this, but can you tell me what table _raw does differently when it comes to triggering per result?

This is a custom alert action that sends an http notification to another system. i need one notification per result in the search (when the search yields more than 10 results). the external system will be utilizing these alerts with source IP information contained in the alert.

Re: realtime alert for each event when reaching x number in y mins

jdinze — Thu, 22 Feb 2018 14:45:02 GMT

i did try that, i think it actually stopped it from working as i am not getting any alerts now.