Handle lag between _indextime and _time

[ThibautB]

Hello,

I figured out some of my alerts didn't trigger because there is a lag between the time of the event and the time the event is indexed, especially with Office 365 logs (and I'm pretty sure the lag comes from Microsoft for a good reason, but that's not the point here)

For example, I have an alert running every 10 minutes and triggering when someone add a forward rule to another mailbox. This alert sometime doesn't trigger because the log is indexed AFTER the search period defined for it.
Concrete example :

indextime               Date                Operation               Rights
2018-10-19 16:08:03 2018-10-19 16:02:20 Add-MailboxPermission   FullAccess
2018-10-19 16:08:03 2018-10-19 16:02:19 Add-RecipientPermission SendAs
2018-10-19 16:03:05 2018-10-19 15:55:42 Add-MailboxPermission   FullAccess
2018-10-19 16:02:05 2018-10-19 15:55:38 Add-MailboxPermission   FullAccess

The first to event did trigger (search between 16h00 and 16h10, event indexed at 16h08) but the last two didn't (search between 15h50 and 16h00, event indexed at 16h02)

Have you got any idea on how to properly handle that other than delaying the search to take the lag in account? Any good idea or feedback would be appreciated.

Thanks!

[valiquet]

To see where your data is the bottle neck use monitoring console.

Use earliest=... and_index_earliest=...

Run the alert every 10 minutes but look at the past 60 minutes and throttle the events
or
Run the alert every 10 minutes and use a sub search or lookup to discard events that already created an alert:

index=... NOT [index=immitable search_name=mySearch | fields uniqueID | format]
index=... NOT [inputlookup ...]

Choose lookup over sub searches since sub searches are not reliable and they have a small max run time and size limit

Best practices make you resilient to ingestion lag and skipped searches. For mission critical work you can be proactive and monitor ingestion and scheduler issues.

[ThibautB]

I'm not sure to understand your last sentence "Best practices make you resilient...", could you develop or link a ressource of what you are talking about ?