topic Re: Splunk Alert Scheduling Issue in Alerting

How to configure Splunk alert scheduling?

zacksoft_wf — Wed, 23 Mar 2022 22:41:37 GMT

My requirement, is to run this alert with a time range of 12 hours and send email twice a day (every 12 hour) based on what it finds.

Here is my configuration,
Cron Expression : * */12 * * *
Time Range: Last 12 hours
Schedule Priority : Default
Schedule Window : 5 minutes

In my local time it runs between 9:30 AM - 10:30 AM and 9:30 PM - 10:30 PM. But, Between those (say between 9:30 AM to 10:30 AM), it triggers multiple emails alerts, like one alert in every 2 min kind of frequency.
What I want is, It should send one email during each run. (i.e. One email after every 12 hours).
Can anyone guide what to change in the scheduling options to achieve this ?

Re: Splunk Alert Scheduling Issue

BahadirS — Tue, 22 Mar 2022 06:30:35 GMT

Hello @zacksoft_wf

Your cron expression schedules your alert every minute 9:00 to 10:00 and 21:00 to 22:00. Your expression would be

30 */12 * * *

I suggest you to check https://crontab.guru/ before scheduling.

to run it once.

Re: Splunk Alert Scheduling Issue

gcusello — Tue, 22 Mar 2022 06:34:40 GMT

Hi @zacksoft_wf,

the solution for your need is the Throttle, that disable your alert for a configurable period after an alert trigger.

So when you save your alert, in addition to the settings you shared, you have to enable throttling for e.g. 2 hours.

In other word you have to:

create your search,
save it as an Alert,
configure the following parameters:
- Alert Type: scheduled
- Time Range: 12 hours
- Cron Expression: * */12 * * *
- Expires: 24 hours
- Trigger Alert when results>0
- Trigger once
- Throttle flagged
- Suppress triggering for 11 hours
- Trigger Actions:
  - Add to triggered alerts
  - Send eMail

Only one hint: I don't like your cron expression, I prefer to define the hors of execution, in other words I'd use:

30 9,21 * * *

in this way, your alert runs at 9.30 and 21.30.

If you want to trigger your alert more times 8every 5 minutes) between 9.30 and 10.30 (AM and PM) but always with the throttle enabled, you could use:

*/5 9,21 * * *

Ciao.

Giuseppe

Re: Splunk Alert Scheduling Issue

zacksoft_wf — Tue, 22 Mar 2022 13:02:11 GMT

Apart from changing the Cron Expression to 30 9,21 * * *
and turning on throttle suppress triggering to 11 hours,
Is there anything else I have to change ?
I am particularly thinking about Schedule Window = 5 Minutes. Should I change it to anything ? What does the Schedule Window option do ?

Re: Splunk Alert Scheduling Issue

gcusello — Tue, 22 Mar 2022 13:03:29 GMT

Hi @zacksoft_wf,

I usually don't use the schedule window parameter.

Ciao.

Giuseppe

Re: Splunk Alert Scheduling Issue

zacksoft_wf — Tue, 22 Mar 2022 13:18:35 GMT

I am really sorry for the confusion.
I couldn't see the "throttle" option, then I realized, what I am looking at is not an 'Alert', but a "Scheduled Report".
Is there a way to suppress the email alerts from a 'Scheduled Report', please ?

But I wonder why did I get so many triggered email for a ScheduledReport. I should get just one at the end of every 12 hour ! Is it because of the 'Scheduling Window' =5 min option that is messing it up ?

Re: Splunk Alert Scheduling Issue

zacksoft_wf — Wed, 23 Mar 2022 12:03:46 GMT

Changing the cron expression to what you suggested sorted out my problem.

Re: Splunk Alert Scheduling Issue

gcusello — Wed, 23 Mar 2022 12:05:09 GMT

Hi @zacksoft_wf ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉