I have built a dashboard for a team who will be monitoring it over a period of several hours daily. They have a response plan for each panel when something pops up. Is there a good method of allowing them to "acknowledge" or "suppress" an individual event within a panel after it's already been taken care of in order to keep the dashboard clean? Each panel refreshes automatically every 10 minutes. We are running Splunk 6.5.3.
Ideally it would clear the event from the dashboard permanently (i.e. subsequent visits to the dashboard will not re-load cleared events), but that isn't a hard requirement if the event remains cleared when the panels automatically refresh.
Depending on the type of data and duration of searches powering these panels, here are a few ideas. They both rely on having (or creating) a field that uniquely identifies a given event. That could be a field that already exists, or you could make it appending together some fields that, when taken together, uniquely characterize an event.
acknowledged="true". Then the searches powering individual panels can search their current data set and also the summary index, cross-apply the acknowledgment using eventstats like
| eventstats values(acknowledged) AS acknowledged BY event_idand then look for only events where
isnull(acknowledged). I'd recommend also integrating some logic that grabs the username of the user who acknowledged the event and the time at which it was acknowledged.
Thanks Elliott! This got the wheels turning.
npapa, I'm trying to do something very similar. If you have any more details on what you came up with I'd love to see it! I'll be digging into this in the mean time.
Instead of lookups I would prefer KV Stores for such use case as it will give better control over individual record to be updated based on action (can be implemented using HTML Dashboard). If folks are monitoring raw events in the dashboard based on the type of event Workflow Actions can be configured