How can I make a dashboard with all triggered alerts?

[swatghare]

Hello All,
I am trying to get the Trigger Alerts dashboard which will show the Alerts which are triggered, I am able to achieve the same using below search query:
index=_audit action=alert_fired ss_app=* | eval ttl=expiration-now() | search ttl>0 | convert ctime(trigger_time) | table trigger_time ss_name severity | rename trigger_time as "Alert Time" ss_name as "Alert Name" severity as "Severity"

However when I click on the alerts listed on dashboard, if ofcourse takes me above query and not the real base search which is written to generate that alert. Does anyone know how do we create dashboard which displays all triggered alerts and if someone click on one specific alert it should display the results / events generated by alerts.

Appreciate quick help.

Regards
Sushant Watghare

[swatghare · Accepted solution]

Hello
Marked this as closed.
I used below query to get the triggered alerts in dashboard:
index=_audit action=alert_fired ss_app=* | eval ttl=expiration-now() | search ttl>0 | convert ctime(trigger_time) | table trigger_time ss_name severity | rename trigger_time as "Alert Time" ss_name as "Alert Name" severity as "Severity"

and then I linked it to API page from where we can drill down the alerts

Regards
Swatghare

[gcusello]

Hi swatghare,
see the "Alert Manager" App at https://splunkbase.splunk.com/app/2665/
You can find useful methods to query triggered alerts.

From this App i used something like this

index=_internal source="/opt/splunk/var/log/splunk/scheduler.log" result_count>0 
| table _time thread_id app savedsearch_name result_count 
| join savedsearch_name [ 
     | rest /services/saved/searches 
     | dedup search 
     | table eai:acl.app  title alert.severity is_scheduled id qualifiedSearch dispatch.earliest_time 
     | rename dispatch.earliest_time AS timerange title AS savedsearch_name eai:acl.app AS app 
     | fields app savedsearch_name id alert.severity timerange qualifiedSearch 
     ]
| lookup alert_severity.csv severity AS alert.severity OUTPUT Severity 
| lookup alert_frequency.csv frequency AS timerange OUTPUT Frequency 
| table app savedsearch_name alert.severity timerange Frequency Severity 

Bye.
Giuseppe

[maciep]

I'm not sure if the search results are available to just be accessed. But in the _audit index, there should be events where action=search and info=granted. The sid from the alert event should match the search_id from that one.

And in that search event, you should have the search itself as well as earliest/latest. So you might be able to build a drilldown to rerun the search over the timeframe of the triggered alert...maybe?