Our web sites seem to alert no matter what, even though it is under the 1000 ms in the macro. Is it alerting based on this search which was cloned from the default website monitoring search? If so, what needs to be changed? It appears that this statement may be at 1-3 milliseconds.
The search that you are using doesn't filter out results that are within the threshold. For alerting, you want a search that only returns results when a problem is identified (zero results when there are zero problems).
The app comes with a search named "website_performance_problems" which is intended for this type of alerting. I recommend using or cloning that one since it filters results unless they exceed the timeout or indicate a failure (like an problematic return code).