Alerting
Highlighted

What are some alerts scheduling methods and best practices?

Engager

Hi Team,

We are trying to reduce the concurrent search count in our environment as upgrading hardware resource is not possible. We have a dedicated alerting which is running about 800 alerts at different frequency. FOR EXAMPLE :

There are around 400 alerts which is running at every 15 min interval defined as such - (*/15 * * * *) in cron meaning the alerts are running at 10 AM,10:15 AM,10:30 etc.

We are planning to schedule the cron job as such -

100  alerts at 10 AM,10:15 AM,10:30 etc. (*/15 * * * *)
100 alerts at 10:1 AM ,10:16 AM,10.31 AM and so on (1/15 * * * *  )
100 alerts at 10:2 AM ,10:17 AM,10.32 AM and so on (2/15 * * * *  )
100 alerts at 10:3 AM ,10:18 AM,10.33 AM and so on (3/15 * * * *  )
100 alerts at 10:4 AM ,10:19 AM,10.34 AM and so on (4/15 * * * *  )

Please suggest if this is a good way of scheduling alerts and do suggest if there are other methods.

Regards

0 Karma
Highlighted

Re: What are some alerts scheduling methods and best practices?

SplunkTrust
SplunkTrust

Spreading schedule searches and alerts across time is a good idea. Here are some other considerations:

When scheduling alerts, consider how long each alert takes to run. It's not helpful to schedule 100 searches each minute if they take longer than a minute to complete.

Don't schedule more alerts at a time than you have CPUs available.

I always recommend setting schedule_window = auto for all scheduled searches.

Review the alerts to determine if they all really need to run every 15 minutes or if some can run less often.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: What are some alerts scheduling methods and best practices?

Engager

Hi Rich,

Thanks for the update. we have 16 core CPU so how many alerts can we configure ?
Yes its only after removing the unwanted alerts, it came down to 400 or something.
So we are trying to come up with a effective solution for this.

Regards.

0 Karma
Highlighted

Re: What are some alerts scheduling methods and best practices?

SplunkTrust
SplunkTrust

Assuming you've not changed the related settings, 16 CPUs can run 22 searches simultaneously. Half of that is for scheduled searches so you can run 11 alerts at a time.

Have you looked in the Monitoring Console for skipped searches? You probably have a lot of them.

---
If this reply helps you, an upvote would be appreciated.
0 Karma