Generalize alerts for detecting performance metric regressions for all type of machines into one

[phoenix_ivy]

Hi Team, I want to consult with you about the following situation:

I setup an email alert for detecting a specific performance metric of one type of machine (config=A). The alert will raise when it detect the latest run value is regressed >5% than the last run value of the same type of machine (config=A).

However, this alert can only detect this for one machine (config=A). If we need to track many other machines (config=A, B, C, D), each one need an alert setup like this since each type of machine's value can only be compared with itself, which is very cumbersome considering we also need to monitor other performance metrics for all machines. 

Do we have a better way to create generalized these alerts into one for this case? Say an alert can loop all type of machines, fetch and compare a specific performance metrics and raise alert accordingly?

[ITWhisperer]

Gather all your stats in one search and get the previous values by config (and metric type?), then do the comparisons. Generate alerts for each result which matches your criteria

[phoenix_ivy]

Thanks! However, gather all stats in one search would merge all configs data. If I compare the last two runs data that came from two different configs, the regression result would be invalid.

[ITWhisperer]

Yes, I said "Gather all your stats in one search and get the previous values by config ..." - this means, use the by clause.