Can you help me fine-tune my malware related alert?

[BrendanCO]

Hello all! I have a pretty simple real-time alert that lets me know if there are any malware related alerts from a syslog feed we have coming in. The alert is:

sourcetype="pan:threat" http_category=malware

Often times, we have a malware alert from a known malicious web site come through like 10 times in a row due to a variety of reasons and Splunk is then performing it's triggered action (in this case, it is forwarding to OpsGenie).

All I'd like to do is to tell Splunk to only forward to OpsGenie one time if there are a bunch of alerts on the same thing within X amount of time. I'm not sure whether to change the trigger conditions from "Per-Result" to something else or to select "Throttling" and adjust that. I've been playing around with both to no avail.

Any help you can provide would be greatly appreciated!

[jlelli]

Something like this?

I set the throttle time to 5 minutes to receive a new alert if the condition is still ongoing; If you need to alert only once to activate the team then you can raise the throttle time to one hour or more according to your preferences and/or policies.

[BrendanCO]

So would it have to get to greater than 10 before a single alert? I need to be able to catch the ones that just come in once as well. What I'm trying to avoid is for the ones that come in 10 times within one minute, to just report on that specific alert once.

[fabiodiogo91]

It depends on the number you defined in the threshold “greater than”. If in 5minutes, you have 11 results for your query, then it triggers the alert. If it matches with only 1 event it will not be triggered.

For your last sentence, the ones coming in more than 10times within a minute, for those, alerts will be suppressed for 5 minutes.

[BrendanCO]

Ok, so is there a way with this feature to say that if the alert is only one time, forward to OpsGenie... and if the alert is greater than 5 in 5 minutes, alert only once to OpsGenie?

[jlelli]

So would it have to get to greater than 10 before a single alert? I need to be able to catch the ones that just come in once as well. What I'm trying to avoid is for the ones that come in 10 times within one minute, to just report on that specific alert once.

If you need to be notified as soon as ONE event comes in then change the alert value "Greater than" to 0.

Ok, so is there a way with this feature to say that if the alert is only one time, forward to OpsGenie... and if the alert is greater than 5 in 5 minutes, alert only once to OpsGenie?

So you need to define two different notification behaviours according to the volume of the alert? I think the easiest way to do so is to clone the alert condition and change the triggers:
- Forward to OpsGenie: is Greater than 0 in 1 minute, trigger "for each result", no throttle (be careful: this setting can create A LOT of notifications)
- Alert to OpsGenie: is greater than 5 in 5 minutes, trigger "Once", throttle 24h (or something like that)

[fabiodiogo91]

Hello,

@jlelli response seems correct to me!

For example, if you have continuous events during those 5minutes, with that throttle you will receive 1 alert every 5minutes.

[jlelli]

For example, if you have continuous events during those 5minutes, with that throttle you will receive 1 alert every 5minutes.

Correct, but only if there are 10+ events again during those 5 minutes, otherwise you would not receive a new alert.

[fabiodiogo91]

Correct, the number of results that you specify in the alert 😃