I don't think you mentioned what version of Splunk you're on but if it's pre-6.3 then the Deployment Monitor app has this build in:
https://splunkbase.splunk.com/app/1294/#/overview
We use a modified version of the alerting that comes with the app, but it's a nice and quick starting point.
If you're 6.3+ then the new built in Deployment Manager has this:
http://docs.splunk.com/Documentation/Splunk/latest/DMC/Platformalerts
I've never used it, so I can't comment on how comparable it is I'm afraid, I'm sure someone else has experience and might be able to comment though.
... View more