Hi Team,
An alert is scheduled to run for every 2 hours
It is getting skipped
per day the alert will run - 12 times
For a week 12*7 = 84 times a week
We could see in the skipped search result that the alert is skipped for 3000 times in last 7 days
How is it possible?
Below search is used to find the skipped search
splunk_server=*prod1-heavy index="_internal" sourcetype="scheduler" host=*-prod1-heavy
| eval scheduled=strftime(scheduled_time, "%Y-%m-%d %H:%M:%S")
| lookup search_env_mapping host AS host OUTPUT tenant
| stats count values(scheduled) as scheduled values(savedsearch_name) as search_name values(status) as status values(reason) as reason values(run_time) as run_time values(dm_node) as dm_node values(sid) as sid by savedsearch_name tenant
| sort -count
| search status!=success
| table scheduled, savedsearch_name, status, reason,count,tenant
