Alerting

What are some alerts scheduling methods and best practices?

keishamtcs
Explorer

Hi Team,

We are trying to reduce the concurrent search count in our environment as upgrading hardware resource is not possible. We have a dedicated alerting which is running about 800 alerts at different frequency. FOR EXAMPLE :

There are around 400 alerts which is running at every 15 min interval defined as such - (*/15 * * * *) in cron meaning the alerts are running at 10 AM,10:15 AM,10:30 etc.

We are planning to schedule the cron job as such -

100  alerts at 10 AM,10:15 AM,10:30 etc. (*/15 * * * *)
100 alerts at 10:1 AM ,10:16 AM,10.31 AM and so on (1/15 * * * *  )
100 alerts at 10:2 AM ,10:17 AM,10.32 AM and so on (2/15 * * * *  )
100 alerts at 10:3 AM ,10:18 AM,10.33 AM and so on (3/15 * * * *  )
100 alerts at 10:4 AM ,10:19 AM,10.34 AM and so on (4/15 * * * *  )

Please suggest if this is a good way of scheduling alerts and do suggest if there are other methods.

Regards

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Spreading schedule searches and alerts across time is a good idea. Here are some other considerations:

When scheduling alerts, consider how long each alert takes to run. It's not helpful to schedule 100 searches each minute if they take longer than a minute to complete.

Don't schedule more alerts at a time than you have CPUs available.

I always recommend setting schedule_window = auto for all scheduled searches.

Review the alerts to determine if they all really need to run every 15 minutes or if some can run less often.

---
If this reply helps you, Karma would be appreciated.
0 Karma

keishamtcs
Explorer

Hi Rich,

Thanks for the update. we have 16 core CPU so how many alerts can we configure ?
Yes its only after removing the unwanted alerts, it came down to 400 or something.
So we are trying to come up with a effective solution for this.

Regards.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Assuming you've not changed the related settings, 16 CPUs can run 22 searches simultaneously. Half of that is for scheduled searches so you can run 11 alerts at a time.

Have you looked in the Monitoring Console for skipped searches? You probably have a lot of them.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...