Alerting

If you have a search that runs every minute and queries the previous minute of events, can events be lost in-between?

Bomo2023
Explorer

I have a scheduled search that runs every minute, querying the previous one minute of time, and alerts if an event is found.

Because the scheduled search interval is the same as the time period being searched, is it theoretically possible for an event to pop up in between the scheduled searches and not get alerted?

If this is a possibility then it feels like the only option would be to search a slightly longer timeframe so that there is some overlap in the searches, but that could mean duplicate events.

The other option would be to use a realtime search rather than a scheduled search., but in general I try to avoid  scheduled realtime searches.

Labels (1)
0 Karma
1 Solution

johnhuang
Motivator

First you need to get a better understanding of the lag time between event time vs ingest/index time. If you run a search before all events have arrived for the search period, you will have missing events.

Run this search to calculate the avg and max ingestion lag time. 

 

index=<YOUR INDEX> earliest=-1h@h latest=-0h@h
| eval ingestion_lag_secs=_indextime-_time
| bucket _time span=10m
| stats count AS event_count avg(ingestion_lag_secs) AS avg_lag_secs min(ingestion_lag_secs) AS min_lag_secs max(ingestion_lag_secs) AS max_lag_secs BY _time
| eval avg_lag_secs=ROUND(avg_lag_secs)
| table _time min_lag_secs max_lag_secs avg_lag_secs event_count

 

Let's assume that all your events arrived within 60 seconds, e.g. lag time of <60s. Then you should offset the latest time in your search to accommodate the lag.

For example, if you have a search that runs every minute and you don't want missing events or have overlapping dupes, then you would set your time modified to:

Solution 1:

 

<YOUR SEARCH> earliest=-2m@m latest=-1m@m

 

 

Solution 2:

Let's assume you have the following requirements:

- High Priority Alert need to be as fast as possible (e.g. <2 minute delay).
- Avoid duplicate events due to overlapping searches
- Avoid missing events due to ingestion lag
- Data have a lag time of 10 seconds to 5 minutes

The approach here would involve getting the last x minutes of data (depending on max lag time) and filtering events based on its the ingestion time.

 

index="<YOUR_SEARCH>" earliest=-5m@m latest=-0m@m
| where _indextime>=relative_time(now(), "-1m@m") AND _indextime<relative_time(now(), "-0m@m")
​

 

 

 

 

View solution in original post

johnhuang
Motivator

First you need to get a better understanding of the lag time between event time vs ingest/index time. If you run a search before all events have arrived for the search period, you will have missing events.

Run this search to calculate the avg and max ingestion lag time. 

 

index=<YOUR INDEX> earliest=-1h@h latest=-0h@h
| eval ingestion_lag_secs=_indextime-_time
| bucket _time span=10m
| stats count AS event_count avg(ingestion_lag_secs) AS avg_lag_secs min(ingestion_lag_secs) AS min_lag_secs max(ingestion_lag_secs) AS max_lag_secs BY _time
| eval avg_lag_secs=ROUND(avg_lag_secs)
| table _time min_lag_secs max_lag_secs avg_lag_secs event_count

 

Let's assume that all your events arrived within 60 seconds, e.g. lag time of <60s. Then you should offset the latest time in your search to accommodate the lag.

For example, if you have a search that runs every minute and you don't want missing events or have overlapping dupes, then you would set your time modified to:

Solution 1:

 

<YOUR SEARCH> earliest=-2m@m latest=-1m@m

 

 

Solution 2:

Let's assume you have the following requirements:

- High Priority Alert need to be as fast as possible (e.g. <2 minute delay).
- Avoid duplicate events due to overlapping searches
- Avoid missing events due to ingestion lag
- Data have a lag time of 10 seconds to 5 minutes

The approach here would involve getting the last x minutes of data (depending on max lag time) and filtering events based on its the ingestion time.

 

index="<YOUR_SEARCH>" earliest=-5m@m latest=-0m@m
| where _indextime>=relative_time(now(), "-1m@m") AND _indextime<relative_time(now(), "-0m@m")
​

 

 

 

 

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...