If you have a search that runs every minute and queries the previous minute of events, can events be lost in-between?

[Bomo2023]

I have a scheduled search that runs every minute, querying the previous one minute of time, and alerts if an event is found.

Because the scheduled search interval is the same as the time period being searched, is it theoretically possible for an event to pop up in between the scheduled searches and not get alerted?

If this is a possibility then it feels like the only option would be to search a slightly longer timeframe so that there is some overlap in the searches, but that could mean duplicate events.

The other option would be to use a realtime search rather than a scheduled search., but in general I try to avoid  scheduled realtime searches.

[johnhuang · Accepted solution]

First you need to get a better understanding of the lag time between event time vs ingest/index time. If you run a search before all events have arrived for the search period, you will have missing events.

Run this search to calculate the avg and max ingestion lag time. 

index=<YOUR INDEX> earliest=-1h@h latest=-0h@h
| eval ingestion_lag_secs=_indextime-_time
| bucket _time span=10m
| stats count AS event_count avg(ingestion_lag_secs) AS avg_lag_secs min(ingestion_lag_secs) AS min_lag_secs max(ingestion_lag_secs) AS max_lag_secs BY _time
| eval avg_lag_secs=ROUND(avg_lag_secs)
| table _time min_lag_secs max_lag_secs avg_lag_secs event_count

Let's assume that all your events arrived within 60 seconds, e.g. lag time of <60s. Then you should offset the latest time in your search to accommodate the lag.

For example, if you have a search that runs every minute and you don't want missing events or have overlapping dupes, then you would set your time modified to:

Solution 1:

<YOUR SEARCH> earliest=-2m@m latest=-1m@m

Solution 2:

Let's assume you have the following requirements:

- High Priority Alert need to be as fast as possible (e.g. <2 minute delay).
- Avoid duplicate events due to overlapping searches
- Avoid missing events due to ingestion lag
- Data have a lag time of 10 seconds to 5 minutes

The approach here would involve getting the last x minutes of data (depending on max lag time) and filtering events based on its the ingestion time.

index="<YOUR_SEARCH>" earliest=-5m@m latest=-0m@m
| where _indextime>=relative_time(now(), "-1m@m") AND _indextime<relative_time(now(), "-0m@m")
​