The icinga alert system (and nagios, generally) has a concept called "macros" that I think would be outstandingly useful in AppD. The idea is pretty simple. Every nagios alert can be passed certain macros that provide information about the alert or the host:
my_alert_handler.sh $HOSTNAME$ $SERVICSTATE$
In the above code, $HOSTNAME$ and $SERVICESTATE$ would map to the alerting host (i.e. 'myserver') and service state (i.e. 'CRITICAL' or 'WARNING'), respectively. Macros let you specify exactly what information an alert or event handler should receive.
Here's the thing: I think it's great that the controller is capable of emailing me, but I don't actually want it to. My production installation uses six different technologies to record performance metrics, three different technologies to alert various members of various teams that something is wrong, and I need *all* of them.
Much better, in my mind, is a custom event handler on the controller. Then I can pass its data to icinga with nsca, or I could increment a counter in tsdb, or graphite, or write a plugin to ganglia, or push them to CloudWatch, or open tickets in my issue tracker. Or I could do all of these things simultaneously.
Right now, custom actions accept a variable number of arguments that are not well defined, and I can't fire test events to find out what an event or policy violation will actually emit. Macro support would obviate all of my trouble, by giving me the freedom to say "this script gets this information; this script gets this other information". It'd turn my controller into a full-fledged alerting system, as opposed to a "monitoring system that coincidentally alerts", and provide instant, simple integration with any number of other technologies in my production installation.
What do you think?
To be clear, I don't think this needs to be terribly complex. I'd settle for "a few more elements inside <action></action> in custom.xml," just as long as I could change it per-custom action, and they were well documented.
