I have an alert set up in my Splunk Enterprise Security environment that is set to trigger when we receive a notable that is marked as either high or critical urgency. This search has worked in the past but did not trigger over the weekend even though a high urgency notable was created.
When I run the search manually over the time range, I can see that it returns results, so this does not appear to be an issue with the search logic. There is also no throttling configured. The search is on a cron schedule to run every 2 minutes and looks over the last 125 seconds of events.
Looking through the audit logs, I can see the following events to confirm that the saved search completed running:
Audit:[timestamp=09-16-2018 13:50:24.206, user=XXXX, action=search, info=completed, search_id='scheduler__admin_YWFtX2FsbF9jdXN0b20tc2VhcmNoZXM__RMD571711d067188a19b_at_1537102200_43245_2C5B5B40-2FE3-4255-A621-60A171C4E9C0', total_run_time=2.63, event_count=0, result_count=0, available_count=0, scan_count=0, drop_count=0, exec_time=1537102203, api_et=1537102075.000000000, api_lt=1537102200.000000000, search_et=1537102075.000000000, search_lt=1537102200.000000000, is_realtime=0, savedsearch_name="New Notable - Alert", search_startup_time="2235", searched_buckets=11, eliminated_buckets=0, considered_events=0, total_slices=0, decompressed_slices=0][n/a]
Audit:[timestamp=09-16-2018 13:50:34.832, user=XXXX, action=search, info=completed, search_id='rsa_scheduler__admin_YWFtX2FsbF9jdXN0b20tc2VhcmNoZXM__RMD571711d067188a19b_at_1537102200_43245_2C5B5B40-2FE3-4255-A621-60A171C4E9C0', total_run_time=2.63, event_count=0, result_count=0, available_count=0, scan_count=0, drop_count=0, exec_time=1537102203, api_et=1537102075.000000000, api_lt=1537102200.000000000, search_et=1537102075.000000000, search_lt=1537102200.000000000, is_realtime=0, savedsearch_name="New Notable - Alert", search_startup_time="2235", searched_buckets=11, eliminated_buckets=0, considered_events=0, total_slices=0, decompressed_slices=0][n/a]
The event that should have triggered an alert occurred at 13:50:31PM on 09-16-18. The second event in the audit log shows that the search was completed three seconds later at 13:50:34PM. Is it possible that these overlapped which prevented the alert from being triggered? The next scheduled search was run at 1:52:24PM so this should have triggered an alert as well.
... View more