Acknowledge/Clear event from dashboard panel

[npapa]

I have built a dashboard for a team who will be monitoring it over a period of several hours daily. They have a response plan for each panel when something pops up. Is there a good method of allowing them to "acknowledge" or "suppress" an individual event within a panel after it's already been taken care of in order to keep the dashboard clean? Each panel refreshes automatically every 10 minutes. We are running Splunk 6.5.3.

Ideally it would clear the event from the dashboard permanently (i.e. subsequent visits to the dashboard will not re-load cleared events), but that isn't a hard requirement if the event remains cleared when the panels automatically refresh.

Thanks!

[elliotproebstel · Accepted solution]

Depending on the type of data and duration of searches powering these panels, here are a few ideas. They both rely on having (or creating) a field that uniquely identifies a given event. That could be a field that already exists, or you could make it appending together some fields that, when taken together, uniquely characterize an event.

1. Build a summary index (or set of summary indexes) supporting the data being displayed. Integrate logic into the dashboard that allows users to select events and submit them to the summary index with an additional field like acknowledged="true". Then the searches powering individual panels can search their current data set and also the summary index, cross-apply the acknowledgment using eventstats like | eventstats values(acknowledged) AS acknowledged BY event_id and then look for only events where isnull(acknowledged). I'd recommend also integrating some logic that grabs the username of the user who acknowledged the event and the time at which it was acknowledged.
2. Create lookups that contain the event_id of acknowledged events, the the time at which the event was acknowledged, and the user who acknowledged it. The search powering a dashboard can then perform a lookup and only display events that are not found in the lookup. The downsides of this approach are that it's pretty easy to wipe out a lookup accidentally, and and your lookup may grow to unmanageable size unless you are routinely trimming out old events.

[niketn]

Instead of lookups I would prefer KV Stores for such use case as it will give better control over individual record to be updated based on action (can be implemented using HTML Dashboard). If folks are monitoring raw events in the dashboard based on the type of event Workflow Actions can be configured

[npapa]

Thanks Elliot, this helped put me in the right direction.

[ineale]

Thanks Elliott! This got the wheels turning.

npapa, I'm trying to do something very similar. If you have any more details on what you came up with I'd love to see it! I'll be digging into this in the mean time.