I've setup several alerts to trigger within my environment and then display the most recent alerts as a search results on a "Review" panel within a dashboard. I would like to have the alerts displayed until a user manually goes in and acknowledges the alert at which point they would move to a second "Acknowledged". Ideally I wanted to set this up using an Event Action that applies only to these alerts and that action would update a tag to say that the event was acknowledged which would make it simple to create my Review and Acknowledged dashboards.
I know workflow actions are limited to links and searches but I wanted to know if anyone knew if something like this was possible within the context of Splunk?
It would also be nice to be able to add a tag or field for the user that acknowledged the alert and when it was acknowledged.
For anyone interested, I was able to create a work around through the use of a look up table. Basically I created a workflow action that saved off the sid and a new 'Status' field with the status Acknowledged. I then limited my search to only those items that were not acknowledged for new events. I'm not sure on the limitations of look up tables in terms of how many events they can hold but this should work for my purposes although I'm running into an issue with trying to execute this workflow action from a dashboard or report. (http://answers.splunk.com/answers/172544/workflow-action-not-working-within-a-dashboard.html)
I wish splunk had 'event ack' feature available 'out of the box'....
I'm currently trying to develop a view which shows critical alerts - some of them are known, and users should be able to move them to another table in the view ' ack'ed events' (adding a comment would be nice as well, but I don't see how that could have been done right now).
@chris2416, would you be able to share more details on your workflow action ?
I do something similar to the following as a search workflow action (after I've created the initial look up table):
index=audit action=alertfired sid=$sid$
| fields + sid
| eval alarmstatus="acknowledged"
| inputlookup append=true alarmlookup
| outputlookup alarm_lookup
This appends the current acknowledged alert to the look up table and then I have two separate search queries, one that searches for all action=alertfired NOT alarmstatus="acknowledge" and a second action=alertfired alarmstatus="acknowledged". This gives me my new errors as well as the acknowledged errors.
I'm not a huge fan of using the look up table and would prefer to have someway to tag events after they've been ingested. The other issue I've run into is the ability to bulk acknowledge, if I suddenly receive 100 alarms I have to acknowledge each one individually.