Generating administrator alerts based on events analysis

[Tim]

I want to build a dashboard that lists alert strings for administrators and creates some basic statistics on these generated alerts. Here's the details.

My event stream comes from a bunch of agents that periodically report values for some key settings. So, I have an index full of daily updates from many agents consisting of:

agentId = , settingA = foo

I want to generate an alert when I detect that settingA has flipped from "foo" to "bar". I'd like to be able to generate a message that says: "Agent X has changed its setting from 'Foo' to 'Bar'." Since I have a large number of events, I need to run this alert generation report as a scheduled background report.

Moreover, I'd like to be able to assign a severity code to each alert message and do a "stats count by severity". In other words, I want to generate some complex stats and charts based on the number of alerts and their severity.

Any recommendations on how to implement this in an application? I know its a broad question but I know someone has implemented something similar.

Thanks, Tim

[bwooden]

If you want to enrich existing data with new values (e.g. severity) you may want to explore lookup files.

Your search can reference an event by a unique field (or combination of fields that would be unique). This unique identifier would be appended to a lookup file along with the new data you wish to associate with it.

You would then use that look-up file when searching through that data to find its associated severity and/or other interesting metadata you've added.

[Tim]

Thanks for the quick reponse. Here's a few thoughts on that approach:

The alert messages would need to be parameterized and would contain data unique to the search result. I think it's possible to dynamically generate lookup files. Any recommendations or best practices on how to do this?

One other consideration: this alert lookup file will become really big and most of the entries are only relevant for 1-2 weeks. It would be better to save the results in the a Splunk index and apply JOINs as needed. Or, I could just append the message text and severity as a field on one of the events.