The built-in email alerting functionality is fairly limited to the degree at which you can customize or control the content of the email itself. There's lots of stuff I'd like to be able to do that I can't right now, for example:
Disk Error on host $results.host$").
That's just off the top of my head. I know I can go hack the
sendemail.py but that's a long-term maintenance nightmare.
Has anybody build some kind of app that augments/hijacks/replaces the out-of-the-box email alerting functionality?
The Splunk app Red Alert could be used as a preprocessor of the data, and then you can easily set up a script that can send out the email in any format that you would like, including HTML email. We aren't using Red Alert this way ourselves, but we are using it to pass the data from the alert to our Event Management System (EMS) for use by the NOC. Red Alert was fairly simple to set up and the programming for the email could be done in any language you desire (except pig-latin).
Splunk have listened.
Version 6.1 of splunk now has TO: CC: & BCC:, Priority, Subject and a multi line Message.
You also have the option of including the search string or not as well as the results.
And they have listed the tokens (like $alert.severity$) that can be used.
And this can be triggered from the search string with the sendmail command.
Email Alerts are much improved - thanks for baking this in! The tokens are a nice touch also.
One thing I think could be added to this is HTML messaging support in the body (message) or footer. It seems to be escaping any HTML characters I add for providing a richer email messing interface.
Example: message="This is an example
Any idea why the default email server under Settings-->Server Settings-->Email Settings-->Mail Host isn't used? It seems to be require that you provide it with each sendemail request which seems odd since there is a default config.
BTW, I did end up making a custom version of the sendemail.py script, and it did make upgrade more complicated. The features I focused on were: 1.) Allowing custom intro text (which came from the "Description" field.) and 2.) I re-arranged the body of the email a bit to move the search link to the bottom (so that message previews would show some of the details. Maybe someday I'll write a full blown app. Of course first I need this to work: http://answers.splunk.com/answers/3110/can-you-add-your-own-custom-alert-action
We've done something similar by writing a scripted alert action script that takes in the results of a search, parse it, then send the email using the script itself.
Beyond that, it might be useful to put in a feature request.
Definitely something that would add value, and I don't think it would be too hard to implement:
Over and above changes to emails, it would make it very easy to configure SMS alerts in splunk, a real value add.