topic Re: Setting alerts on exceptions in Alerting

Setting alerts on exceptions

usmanchaudhri — Thu, 10 Feb 2011 02:06:07 GMT

Hi,

I potentially want to set a scheduled search - where i specify the list of exceptions in the search - and if there is any new exception outside of those listed exceptions, Splunk should send an email alert.

For example: Consider, here is my list of exceptions: "error: null pointer exception (login.class:1494)" "error: database down exception (database.class:1594)" "error: read PFD (readPDF.class:1694)"

Now, whenever there is a new exception generated (outside of those listed above), Splunk sends me alert.

Thanks for looking into this. Usman Chaudhri

Re: Setting alerts on exceptions

David — Thu, 10 Feb 2011 05:11:36 GMT

The way I did this was to set up eventtypes.conf so that each event has an eventtype. You can then run a query:

YourSearch NOT eventtype=*

And that will show all the events that aren't on your pre-defined list. You can toss that in a scheduled search no problem.

http://www.splunk.com/base/Documentation/latest/Admin/eventtypesconf

Re: Setting alerts on exceptions

usmanchaudhri — Thu, 10 Feb 2011 07:40:35 GMT

Yeah, that's what i ended up doing. I saved a search as an eventtype, the search had the pre-defined list of events. Than i went ahead and scheduled another search and just specified eventtype!=< predefine list >. This gave me list of new events.

Thank you
Usman Chaudhri

Re: Setting alerts on exceptions

David — Thu, 10 Feb 2011 08:17:46 GMT

That sounds like a good solution. For my use case, we had a relatively small number of events (45 or so) with fairly unique descriptors. Having a specific event type to each error message also allows me to toss up a dashboard with the daily average over the last month for each event type, compared with the last 24 hours. That way I can catch changes in known errors. Food for thought 😉