This Microsoft article https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/reference-reports-latencies#security-reports talks about the latency for sign-ins and audit logs in Azure. The latency is between 2 to 5 mins. My understand would be that the logs will be available in Azure portal (also ready for the API to pull) within 5 mins of the originating event. So I think setting the polling frequency to >300s should be OK. However, I have concern about this Add-on using the largest siginDateTime/activityDateTime seen during the query as the checkpoint timestamp. My reasoning is that Azure logs may come in different order, and we will miss some events came in late but their originating event timestamps are before the checkpoint.
I have the following scenario in mind:
My Signins Input starts at 1:10pm (with polling interval 10 mins) and the current checkpoint is 1:00pm
1st input/query ran and pulling logs from 1:00pm to 1:10pm. The Add-on set the largest siginDataTime as the checkpoint. (Let’s say the largest signin time seen from the query is 1:07pm, now the checkpoint is 1:07pm)
If I have a originating sign-in event happened at 1:06pm but this log is not made available until 1:11pm (5 mins delay). So my 1st query that ran at 1:10pm missed this log and that’s OK as I will expect the next query will pick it up.
Now at 1:20pm my 2nd input ran. This query however just pulled log from 1:07pm (current checkpoint) to 1:20pm. At this point, my 1:06pm sign-in event is going to be skipped.
As suggested by jconger, the "Azure Monitor Add-on for Splunk" may be the better way to collect near real time from the an Event Hub.
FYI... I have been trying to collect Azure AD logs (sign-in, audit), Azure AD risk events, as well as Office 365 logs into Splunk. I feel in general the latencies in the Microsoft reporting infrastructure causing lots of confusion/issues on how we can properly schedule our data ingestion without incomplete/duplicate data problem. It makes it harder to use the data for near real time monitoring/reporting solution.
... View more