Alerting

Staggering cron alerts?

kwkeefer
Explorer

After nearly doubling the amount of scheduled (cron) alerts in my Splunk environment, I'm starting to see some performance issues.

The alerts run every five minutes, and look at the previous five minute's worth of data.

earliest = -5m@m 
latest = now
cron expression: */5 * * * *

It has been recommended to me to stagger the scheduled alerts so that, for example, some are running at 12:01, others run at 12:02, others run at 12:03, etc.

Is this possible? I'm having difficulty finding an options in the 'edit alert' page to further fine tune the cron schedule so that I can set an actual start time.

Also, the advice I got is perplexing, because I added the alerts manually; I would assume they would already be staggered because of the fact that I didn’t wait until exactly 12:00 or 12:05 to hit the submit button when creating each alert. Is that an incorrect assumption?

0 Karma

woodcock
Esteemed Legend

You should just set Schedule window to Auto and leave it at */5 and Splunk will do the staggering for you.

adonio
Ultra Champion

hello there,
first i will recommend that if you have the alert run every 5 minutes, let it search from 6 or 7 minutes ago until 1 or 2 minutes ago or go further 15 -10 minutes ago. that will give you better search performance then searching till time=now
here is how to manually edit cron to whichever expression you would like under the save alert popoup:

alt text

alt text

DalJeanis
SplunkTrust
SplunkTrust

Also, in answer to OPs question about staggering the times...

cron expression: 1/5 * * * * ... every 5 minutes starting at 1 minute after the hour

cron expression: 2/5 * * * * ... every 5 minutes starting at 2 minutes after the hour

cron expression: 3/5 * * * * ... every 5 minutes starting at 3 minutes after the hour

0 Karma

kwkeefer
Explorer

Would it be something like

1/5* * * *

or just

1/5

Sorry - I am not very experienced at cron / splunk.

0 Karma

adonio
Ultra Champion

this tool explains it better then i do
https://crontab.guru/

0 Karma

kwkeefer
Explorer

Thank you.

0 Karma

kwkeefer
Explorer

First, thanks for your response.

You don't by chance know of any documentation that mentions why using the 'now' option is better for performance, do you? I saw in 'Alert scheduling tips' in the docs said that delaying the search will help ensure you capture all the results, but it doesn't mention performance. I'm curious to know more.

0 Karma
Get Updates on the Splunk Community!

New Cloud Intrusion Detection System Add-on for Splunk

In July 2022 Splunk released the Cloud IDS add-on which expanded Splunk capabilities in security and data ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...