I know this seems obvious I'm searching 5 minutes back and alerting on the results every 1 minute so there is 4 minutes of over lap on each search. But due to some internal issues the logs are not always indexed right on time so I can't to a 1 minute search for a 1 minute alert or I would for sure miss stuff.
The alert is throttled to to suppress triggering for 5 minutes but this is missing alerts too. Is there any way for the alert to be aware of a previous alert result and make a dynamic allow list?
Alerts can be throttled by field values. Edit the alert and change the Trigger setting to "For each result". That will show the "Suppress results containing field value" box. Enter the names of the fields (separated by commas) you want Splunk to use to throttle the alerts. If the alert triggers again with the given time and the field(s) listed have values already alerted then no new alert will be sent.