Alerting

Staggering cron alerts?

kwkeefer
Explorer

After nearly doubling the amount of scheduled (cron) alerts in my Splunk environment, I'm starting to see some performance issues.

The alerts run every five minutes, and look at the previous five minute's worth of data.

earliest = -5m@m 
latest = now
cron expression: */5 * * * *

It has been recommended to me to stagger the scheduled alerts so that, for example, some are running at 12:01, others run at 12:02, others run at 12:03, etc.

Is this possible? I'm having difficulty finding an options in the 'edit alert' page to further fine tune the cron schedule so that I can set an actual start time.

Also, the advice I got is perplexing, because I added the alerts manually; I would assume they would already be staggered because of the fact that I didn’t wait until exactly 12:00 or 12:05 to hit the submit button when creating each alert. Is that an incorrect assumption?

0 Karma

woodcock
Esteemed Legend

You should just set Schedule window to Auto and leave it at */5 and Splunk will do the staggering for you.

adonio
Ultra Champion

hello there,
first i will recommend that if you have the alert run every 5 minutes, let it search from 6 or 7 minutes ago until 1 or 2 minutes ago or go further 15 -10 minutes ago. that will give you better search performance then searching till time=now
here is how to manually edit cron to whichever expression you would like under the save alert popoup:

alt text

alt text

DalJeanis
SplunkTrust
SplunkTrust

Also, in answer to OPs question about staggering the times...

cron expression: 1/5 * * * * ... every 5 minutes starting at 1 minute after the hour

cron expression: 2/5 * * * * ... every 5 minutes starting at 2 minutes after the hour

cron expression: 3/5 * * * * ... every 5 minutes starting at 3 minutes after the hour

0 Karma

kwkeefer
Explorer

Would it be something like

1/5* * * *

or just

1/5

Sorry - I am not very experienced at cron / splunk.

0 Karma

adonio
Ultra Champion

this tool explains it better then i do
https://crontab.guru/

0 Karma

kwkeefer
Explorer

Thank you.

0 Karma

kwkeefer
Explorer

First, thanks for your response.

You don't by chance know of any documentation that mentions why using the 'now' option is better for performance, do you? I saw in 'Alert scheduling tips' in the docs said that delaying the search will help ensure you capture all the results, but it doesn't mention performance. I'm curious to know more.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...