topic Re: Splunk Alerts suppression issue? in Alerting

Splunk Alerts suppression issue?

L1mLam — Wed, 21 Jul 2021 12:52:50 GMT

We have a Splunk Alert set up with the following configuration:

SETTINGS
Alert type = Scheduled (Run on Cron Schedule)
Time Range = Today
Cron Expression = *****
Expires = 24 hours

TRIGGER CONDITIONS
Trigger alert when = Number of Results > 0
Trigger = Once
Throttle = Ticked
Suppress triggering for = 1 day

TRIGGER ACTIONS
When triggered
- Add to Triggered Alerts
- Send email

The issue that we are experiencing is that if we have 3 events occur at different times throughout the day, we are only receiving an email for the first one. Also, the following day (within the 24 hour period from the previous alert) we are not receiving any email notifications. In all cases if I select the Splunk Alert and view the results I see all the events shown here, including those for which no email notification was received..

I believe the issue here has to do with the following settings:

Trigger = Once
Throttle = Ticked
Suppress triggering for = 1 day

From the Splunk documentation it is not clear whether all Splunk alerts would get suppressed after the first one, or just repeated Splunk Alerts for the same event. I am assuming that it's the former as this would explain why we don't see any further email notifications until the 1 day / 24 hour period expires(?)

I think changing the settings to the following:

Trigger = For each result
Throttle = Ticked
Suppress triggering for = 1 day

Will at least mean that we receive only one event in each email notification (for simultaneous alerts ... another issue that exists) but will not fix the suppressed email notifications. Furthermore, removing the Throttle seems to just continuously alert on the same event.

I want to keep the "Scheduled Alert" type (rather than "Realtime") due to the set-up that we have here and also I am unable to play around too much with the configuration in test as we do not have email notifications in this environment (only in our live environment).

The goal, in case it's not yet clear from the above, is to receive a single email notification for each event. Can you please advise / suggest the correct change that I should make to achieve this?

Re: Splunk Alerts suppression issue?

richgalloway — Wed, 21 Jul 2021 15:24:35 GMT

Throttling a once-per-search alert prevents it from triggering again until the throttle period expires. If you switch the alert to one-per-event then you can select a field (host, for example) on which to base the throttling. New alerts for the same field value will not trigger, but a different value will trigger an alert.

Re: Splunk Alerts suppression issue?

L1mLam — Mon, 26 Jul 2021 15:00:40 GMT

Apologies for the delayed response ... due to technical issues logging into my Splunk account.

Thank you for your suggestion. I have now accepted this and made some adjustments also to the cron schedule (from **** to */5 * * * *), with the addition of a "Suppress triggering for 600 seconds" (i.e. 5 mins) added in too.

I am trialling this solution over this next week to see if any of our (infrequent) alert events result in this Splunk Alert correctly being triggered.

Re: Splunk Alerts suppression issue?

L1mLam — Mon, 26 Jul 2021 15:01:30 GMT

*suggestion @richgalloway

Re: Splunk Alerts suppression issue?

richgalloway — Mon, 26 Jul 2021 15:22:23 GMT

FWIW, 600 seconds is 10 minutes, not 5.

You say you accepted an answer, but no answer is so marked.

Re: Splunk Alerts suppression issue?

L1mLam — Mon, 26 Jul 2021 15:42:51 GMT

@richgalloway that's a very good point! 600 secs is 10 mins - apologies for the confusion.

I used that example as someone provided this to me within my organisation but you're right that this should be set to 300 for 5 mins. Incidentally, for the suppression period I don't think that it makes much difference as both values should ensure that the next alert is outside of the 5 minute cron schedule checkpoint window.

I didn't yet "Accept as Solution" until I have validated this in my test environment, as alluded in my earlier comment.