There's a really cool blog up on the Splunk Blogs that talks about alerts from a data science view. It's really cool to see how alerting is aggregated and delivered:
A Deep Dive into SignalFx Microservices APM Alerts
Would like to hear your thoughts on this.
davemc