Alerting

What are some alerts scheduling methods and best practices?

keishamtcs
Engager

Hi Team,

We are trying to reduce the concurrent search count in our environment as upgrading hardware resource is not possible. We have a dedicated alerting which is running about 800 alerts at different frequency. FOR EXAMPLE :

There are around 400 alerts which is running at every 15 min interval defined as such - (*/15 * * * *) in cron meaning the alerts are running at 10 AM,10:15 AM,10:30 etc.

We are planning to schedule the cron job as such -

100  alerts at 10 AM,10:15 AM,10:30 etc. (*/15 * * * *)
100 alerts at 10:1 AM ,10:16 AM,10.31 AM and so on (1/15 * * * *  )
100 alerts at 10:2 AM ,10:17 AM,10.32 AM and so on (2/15 * * * *  )
100 alerts at 10:3 AM ,10:18 AM,10.33 AM and so on (3/15 * * * *  )
100 alerts at 10:4 AM ,10:19 AM,10.34 AM and so on (4/15 * * * *  )

Please suggest if this is a good way of scheduling alerts and do suggest if there are other methods.

Regards

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Spreading schedule searches and alerts across time is a good idea. Here are some other considerations:

When scheduling alerts, consider how long each alert takes to run. It's not helpful to schedule 100 searches each minute if they take longer than a minute to complete.

Don't schedule more alerts at a time than you have CPUs available.

I always recommend setting schedule_window = auto for all scheduled searches.

Review the alerts to determine if they all really need to run every 15 minutes or if some can run less often.

---
If this reply helps you, Karma would be appreciated.
0 Karma

keishamtcs
Engager

Hi Rich,

Thanks for the update. we have 16 core CPU so how many alerts can we configure ?
Yes its only after removing the unwanted alerts, it came down to 400 or something.
So we are trying to come up with a effective solution for this.

Regards.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Assuming you've not changed the related settings, 16 CPUs can run 22 searches simultaneously. Half of that is for scheduled searches so you can run 11 alerts at a time.

Have you looked in the Monitoring Console for skipped searches? You probably have a lot of them.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...