I currently have some alerts being triggered when they shouldn't be. The search is performing a host alive check, where the host is sending an event every second. The search is looking over a period of 1 minute, so it should be returning 60 events. An email will be sent if the event count is < 50. This search is scheduled to run as a cron job, to run every minute
The time frame of the alert is -3@m to -2@m, this was set to make sure there wasn't and issue with searching most recent event.
Randomly the alert will be triggered, but when I view the alert, 60 events are shown in the search. Also when I run the search manually, 60 events are returned.
Looking into this further, I have looked in _audit for events related to the specific search. What I have noticed is that when the alert is triggered, the search results return nothing.
Sample of normal result:
action=search, info=completed, search_id=<SEARCH_ID>, total_run_time=0.46, event_count=60, result_count=60, avaliable_count=60, scan_count=60, drop_count=60
When an alert is triggered:
action=search, info=completed, search_id=<SEARCH_ID>, total_run_time=0.45, event_count=0, result_count=0, avaliable_count=0, scan_count=0, drop_count=0
I can't work out why the search is returning 0 results. To me it appears as if the search didn't run or was unable to run correctly.
... View more