I want to build a dashboard that lists alert strings for administrators and creates some basic statistics on these generated alerts. Here's the details.
My event stream comes from a bunch of agents that periodically report values for some key settings. So, I have an index full of daily updates from many agents consisting of:
agentId = , settingA = foo
I want to generate an alert when I detect that settingA has flipped from "foo" to "bar". I'd like to be able to generate a message that says: "Agent X has changed its setting from 'Foo' to 'Bar'." Since I have a large number of events, I need to run this alert generation report as a scheduled background report.
Moreover, I'd like to be able to assign a severity code to each alert message and do a "stats count by severity". In other words, I want to generate some complex stats and charts based on the number of alerts and their severity.
Any recommendations on how to implement this in an application? I know its a broad question but I know someone has implemented something similar.
If you want to enrich existing data with new values (e.g. severity) you may want to explore lookup files.
Your search can reference an event by a unique field (or combination of fields that would be unique). This unique identifier would be appended to a lookup file along with the new data you wish to associate with it.
You would then use that look-up file when searching through that data to find its associated severity and/or other interesting metadata you've added.
Thanks for the quick reponse. Here's a few thoughts on that approach:
The alert messages would need to be parameterized and would contain data unique to the search result. I think it's possible to dynamically generate lookup files. Any recommendations or best practices on how to do this?
One other consideration: this alert lookup file will become really big and most of the entries are only relevant for 1-2 weeks. It would be better to save the results in the a Splunk index and apply JOINs as needed. Or, I could just append the message text and severity as a field on one of the events.