Send an alert if one event doesn't occur in 10 min

[Anud]

How Send an alert if one event doesn't occur in 10 min with below format data.
The data will send every 1 hour with 30mins interval.

example:  alert has trigger for the below data is 2:40

_time	ID	Bill_ID
2024-01-12T03:10:53.000-06:00	TTF5	80124
2024-01-12T03:08:07.000-06:00	TFB6	84958
2024-01-12T02:34:54.000-06:00	TFB6	84958
2024-01-12T02:09:48.000-06:00	TTF5	80124
2024-01-12T02:07:02.000-06:00	TFB6	84958
2024-01-12T01:36:59.000-06:00	TTF5	80124
2024-01-12T01:33:37.000-06:00	TFB6	84958
2024-01-12T01:11:13.000-06:00	TTF5	80124
2024-01-12T01:07:22.000-06:00	TFB6	84958
2024-01-12T00:37:08.000-06:00	TTF5	80124
2024-01-12T00:35:08.000-06:00	TFB6	84958
2024-01-12T00:11:16.000-06:00	TTF5	80124
2024-01-12T00:10:20.000-06:00	TFB6	84958
2024-01-11T23:36:19.000-06:00	TTF5	80124
2024-01-11T23:34:17.000-06:00	TFB6	84958

[PickleRick]

OK. Answer to this question is not that straightforward. But often the original question might not be exactly what you need to get from your data.

Anyway.

The first and, let's be honest, worst idea, would be to create a real-time alert with a window of 10 minutes and trigger an alert when there are no results for your search. But this is a very bad idea! Using real-time searches (and alerts) is generally a bad practice since they hog up resources.

So you should be checking for the results returned by historical searches. You should look at your data "backwards" and verify whether there was an event when there should be one.

Now, your wording is a bit confusing and ambiguous.

Firstly, what does "every 1 hour with 30 minutes interval" mean? Either you do something every 1 hour or with 30 minutes interval (which means every half an hour). Depending on that you should schedule your backward-looking search so that it finds the proper data if it's been ingested, is a bit late vs. the event time so that you can afford for some degree of latency in ingestion process (especially that you seem to be ingesting the events in batches) and need to take care in order to not overlap your results but on the other hand, since you're searching for the time difference between events, you're not "losing" any base events in your search.

@gcusellotries (but it won't work properly - the timechart will fill missing 10-minute segments with values of 0 so the overall count will be just a count of the 10-minute segments within the search timerange _if there is at least one event during that time) to align your events into 10 minute buckets. The better approach here would be

index=your_index
| timechart count span=10m
| where count=0

This way you'll get a list of 10 minute segments during which you didn't get event a single event. But it's not exactly what you asked for because if you get one event at 1:11AM and another at 1:28AM, they are 17 minutes apart but they are counted in separate 10-minute segments so both those segments are "ok".

The way to calculate "lag" between events would be to carry over the _time value from the previous event using autoregress command (or streamstats; streamstats is more versatile but harder to use; in simple case, autoregress is easier and more straightforward)

index=<your_index>
| autoregress _time as oldtime
| eval eventlag=oldtime-_time

This way you get a field called "eventlag" which tells you how much time the event was before/after previous/next one (I never remember which direction the values are copied so you need to test it and possibly reverse the eventlag calculation to (_time-oldtime) in order to not get negative values.

This way you can find in which moment your eventlag was bigger than 600 (the _time is expressed in seconds).

[Anud]

Thanks for response @PickleRick 

Here my logic is for every 30mins one event1 will generate and within 5mins another event2 has to generate.
If not it has to trigger the alert.

Hope you understand the logic.

[PickleRick]

The problem here is in deciding on the proper logic for the alert.

If you're supposed to get an event at the same time of the hour (like around XX:32) and then next one within 5 minutes, you can get away with scheduling a search at - for example each XX:08 and XX:38 and searching some 6-7 minutes into the past and checking if you have less than two results. That's the simplest solution and can often be enough.

But if your case is more complicated (like the time the events are generated is more "floating" around the hour), you might need to schedule a search more often, search through data some 30+ minutes back and calculate the event lag as I mentioned.

It's about defining a problem precisely 🙂

[inventsekar]

Hi @Anud 
>>> example:  alert has trigger for the below data is 2:40

on your data list, i am not sure to find out where is the 2:40 at all. 

ok, lets do like this. simply create a search for a condition(either ID or Bill_ID) and count the results. 

if you have results equal to zero, then on the alert condition, add your trigger condition. 

Pls let me/us know if any questions, thanks. 

PS - if any reply helped you, pls upvote/add karma points. if any reply solves your query, pls accept it as solution, thanks. 

[Anud]

Thanks for response.

For using eval condition and count the values not giving proper results. 

Here my logic is for every 30mins one event1 will generate and within 5mins another event2 has to generate.
If not it has to trigger the alert.

[gcusello]

Hi @Anud,

let me understand: you have data every hour but you want to trigger an alert if there's a delay more than 10 minutes between events, is this correct?

In this case, you could run something like this:

index=your_index
| timechart count span=10m
| stats count AS checks
| where checks<6

you should run this alert every hour to have 6 checks every hour.

Ciao.

Giuseppe