Alerting

If you have a search that runs every minute and queries the previous minute of events, can events be lost in-between?

Bomo2023
Explorer

I have a scheduled search that runs every minute, querying the previous one minute of time, and alerts if an event is found.

Because the scheduled search interval is the same as the time period being searched, is it theoretically possible for an event to pop up in between the scheduled searches and not get alerted?

If this is a possibility then it feels like the only option would be to search a slightly longer timeframe so that there is some overlap in the searches, but that could mean duplicate events.

The other option would be to use a realtime search rather than a scheduled search., but in general I try to avoid  scheduled realtime searches.

Labels (1)
0 Karma
1 Solution

johnhuang
Motivator

First you need to get a better understanding of the lag time between event time vs ingest/index time. If you run a search before all events have arrived for the search period, you will have missing events.

Run this search to calculate the avg and max ingestion lag time. 

 

index=<YOUR INDEX> earliest=-1h@h latest=-0h@h
| eval ingestion_lag_secs=_indextime-_time
| bucket _time span=10m
| stats count AS event_count avg(ingestion_lag_secs) AS avg_lag_secs min(ingestion_lag_secs) AS min_lag_secs max(ingestion_lag_secs) AS max_lag_secs BY _time
| eval avg_lag_secs=ROUND(avg_lag_secs)
| table _time min_lag_secs max_lag_secs avg_lag_secs event_count

 

Let's assume that all your events arrived within 60 seconds, e.g. lag time of <60s. Then you should offset the latest time in your search to accommodate the lag.

For example, if you have a search that runs every minute and you don't want missing events or have overlapping dupes, then you would set your time modified to:

Solution 1:

 

<YOUR SEARCH> earliest=-2m@m latest=-1m@m

 

 

Solution 2:

Let's assume you have the following requirements:

- High Priority Alert need to be as fast as possible (e.g. <2 minute delay).
- Avoid duplicate events due to overlapping searches
- Avoid missing events due to ingestion lag
- Data have a lag time of 10 seconds to 5 minutes

The approach here would involve getting the last x minutes of data (depending on max lag time) and filtering events based on its the ingestion time.

 

index="<YOUR_SEARCH>" earliest=-5m@m latest=-0m@m
| where _indextime>=relative_time(now(), "-1m@m") AND _indextime<relative_time(now(), "-0m@m")
​

 

 

 

 

View solution in original post

johnhuang
Motivator

First you need to get a better understanding of the lag time between event time vs ingest/index time. If you run a search before all events have arrived for the search period, you will have missing events.

Run this search to calculate the avg and max ingestion lag time. 

 

index=<YOUR INDEX> earliest=-1h@h latest=-0h@h
| eval ingestion_lag_secs=_indextime-_time
| bucket _time span=10m
| stats count AS event_count avg(ingestion_lag_secs) AS avg_lag_secs min(ingestion_lag_secs) AS min_lag_secs max(ingestion_lag_secs) AS max_lag_secs BY _time
| eval avg_lag_secs=ROUND(avg_lag_secs)
| table _time min_lag_secs max_lag_secs avg_lag_secs event_count

 

Let's assume that all your events arrived within 60 seconds, e.g. lag time of <60s. Then you should offset the latest time in your search to accommodate the lag.

For example, if you have a search that runs every minute and you don't want missing events or have overlapping dupes, then you would set your time modified to:

Solution 1:

 

<YOUR SEARCH> earliest=-2m@m latest=-1m@m

 

 

Solution 2:

Let's assume you have the following requirements:

- High Priority Alert need to be as fast as possible (e.g. <2 minute delay).
- Avoid duplicate events due to overlapping searches
- Avoid missing events due to ingestion lag
- Data have a lag time of 10 seconds to 5 minutes

The approach here would involve getting the last x minutes of data (depending on max lag time) and filtering events based on its the ingestion time.

 

index="<YOUR_SEARCH>" earliest=-5m@m latest=-0m@m
| where _indextime>=relative_time(now(), "-1m@m") AND _indextime<relative_time(now(), "-0m@m")
​

 

 

 

 

Get Updates on the Splunk Community!

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Your Voice Matters! Help Us Shape the New Splunk Lantern Experience

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Building Momentum: Splunk Developer Program at .conf25

At Splunk, developers are at the heart of innovation. That’s why this year at .conf25, we officially launched ...