topic Re: What are some alerts scheduling methods and best practices? in Alerting

What are some alerts scheduling methods and best practices?

keishamtcs — Thu, 27 Jun 2019 16:35:38 GMT

Hi Team,

We are trying to reduce the concurrent search count in our environment as upgrading hardware resource is not possible. We have a dedicated alerting which is running about 800 alerts at different frequency. FOR EXAMPLE :

There are around 400 alerts which is running at every 15 min interval defined as such - (*/15 * * * *) in cron meaning the alerts are running at 10 AM,10:15 AM,10:30 etc.

We are planning to schedule the cron job as such -

100  alerts at 10 AM,10:15 AM,10:30 etc. (*/15 * * * *)
100 alerts at 10:1 AM ,10:16 AM,10.31 AM and so on (1/15 * * * *  )
100 alerts at 10:2 AM ,10:17 AM,10.32 AM and so on (2/15 * * * *  )
100 alerts at 10:3 AM ,10:18 AM,10.33 AM and so on (3/15 * * * *  )
100 alerts at 10:4 AM ,10:19 AM,10.34 AM and so on (4/15 * * * *  )

Please suggest if this is a good way of scheduling alerts and do suggest if there are other methods.

Regards

Re: What are some alerts scheduling methods and best practices?

richgalloway — Thu, 27 Jun 2019 18:07:31 GMT

Spreading schedule searches and alerts across time is a good idea. Here are some other considerations:

When scheduling alerts, consider how long each alert takes to run. It's not helpful to schedule 100 searches each minute if they take longer than a minute to complete.

Don't schedule more alerts at a time than you have CPUs available.

I always recommend setting schedule_window = auto for all scheduled searches.

Review the alerts to determine if they all really need to run every 15 minutes or if some can run less often.

Re: What are some alerts scheduling methods and best practices?

keishamtcs — Thu, 27 Jun 2019 18:16:05 GMT

Hi Rich,

Thanks for the update. we have 16 core CPU so how many alerts can we configure ?
Yes its only after removing the unwanted alerts, it came down to 400 or something.
So we are trying to come up with a effective solution for this.

Regards.

Re: What are some alerts scheduling methods and best practices?

richgalloway — Thu, 27 Jun 2019 20:20:18 GMT

Assuming you've not changed the related settings, 16 CPUs can run 22 searches simultaneously. Half of that is for scheduled searches so you can run 11 alerts at a time.

Have you looked in the Monitoring Console for skipped searches? You probably have a lot of them.