Good search for alerts that haven't triggered in at least a month?

[aberkow]

I basically want to audit the many dozen infrastructure alerts we have to see which ones aren't firing (this doesn't necessarily mean somethings wrong, but is good information to have in a digest). Anyone have a search they use for this or a similar idea?

[woodcock · Accepted solution]

You may have to extend index retention or artifact TTL but here are your options:

index="_audit" action="alert_fired"

OR:

|rest/servicesNS/-/-/alerts/fired_alerts

OR (you can see alert_actions here):

index="_internal" sourcetype="scheduler" thread_id="*" app="*"

[aberkow]

Building off of @woodcock's suggestion, ended up with something like this, still thinking about the where clause or triggering condition though:

index="_audit" action="alert_fired" earliest=-13d user={userYouCareAbout1 OR userYouCareAbout2}
| bin _time span=7d as week 
| stats count by action, ss_name, week 
| eventstats max(week) as latestWeek, min(week) as earliestWeek 
| eval latestWeek=if(week=latestWeek, count, null()), 
       earliestWeek=if(week=earliestWeek,count, null())
| stats values(earliestWeek) as earliestWeekCount, values(latestWeek) as latestWeekCount by ss_name

[gjanders]

In Alerts for Splunk Admins I did (github link):
SearchHeadLevel - Alerts that have not fired an action in X days

However the audit logs might be more accurate as I haven't used this query in years...

index=_internal source="*scheduler.log" sourcetype=scheduler `searchheadhosts` alert_actions!="" 
| rex ", app=\"(?P<app>[^\"]+)\","
| stats count by savedsearch_name, app 
| append 
    [| rest splunk_server=local /servicesNS/-/-/saved/searches 
    | search actions!="summary_index" actions!="" next_scheduled_time!="" search!="| noop" 
    | table eai:acl.app, title 
    | eval fromRESTQuery=""
    | rename title as savedsearch_name, eai:acl.app as app ]
| eventstats count(eval(isnotnull(fromRESTQuery))) AS restCount, count by savedsearch_name, app
| where restCount=1 AND count=1
| table savedsearch_name, app