topic How do I make an alert which triggers if an event's delay is unusually long? in Splunk Search

How do I make an alert which triggers if an event's delay is unusually long?

matthewg — Tue, 29 Sep 2020 21:36:04 GMT

I have multiple events such as below:

Key points here:

New values of event_type may be added randomly and the schedule may be randomly changed for different event types.
- I just want an alert for if an event gets missed or delayed more than x (seconds/minutes/hours) longer than between the previous events
- Then I want it to see the new normal for that event type.
- Specifically: if the difference in time between now and the last event is more than x (seconds/minutes/hours) greater than the difference between the last event and the previous one, I want to trigger an alert.
8:03am event_type=1
8:05am event_type=2
8:15am event_type=2
8:25am event_type=2
8:33am event_type=1
9:15am event_type=2
10:05am event_type=2
Example: The search should see that event type=1 happened at 8:03 and at 8:33 (difference 30 min) so the next event_type=1 should be at 9:03. If x is 5 min., then at 9:08, an alert should be triggered if event_type=1 is not seen.

Similarly, event type=2 happened at 8:15 and 8:25 (difference 10 min) so if x is 5 min then at 8:40 an alert should be triggered for a missing event_type=2.

In this example someone changed the schedule for event_type=2 (to 50 minutes), so after seeing an event at 9:15, it should expect the next one at 10:05 since the difference between the last two was 50 minutes.

What I tried:

event_type="*" | streamstats current=f window=2 earliest(_time) as time_2 latest(_time) as time_1 by event_type| eval time_0_diff = time_1 - _time  | eval time_1_diff=time_2 - time_1 | eval time_variance=time_0_diff - time_1_diff |

This only works if I have an event though and I am specifically looking for when an event is missing (or delayed).

Re: How do I make an alert which triggers if an event's delay is unusually long?

renjith_nair — Sun, 14 Oct 2018 13:28:13 GMT

@matthewg,

Try

event_type="*"
|sort event_type,- _time 
|streamstats last(_time) as prev_time by event_type current=f window=1|eval diff=round((prev_time-_time)/60,0)
|streamstats count as rowno,latest(_time) as latest_time by event_type|where rowno==2 
|eval latest_diff=round((now()-latest_time)/60,0)
|where latest_diff>diff

    |sort event_type,- _time 
    |streamstats last(_time) as prev_time by event_type current=f window=1|eval diff=round((prev_time-_time)/60,0)

Sorted event_type in ascending and _time in descending so that the similar events are adjacent and sorted in the order of _time(latest first). Then calculate the difference between the current latest and second latest in minutes

    |streamstats count as rowno,latest(_time) as latest_time by event_type|where rowno==2

Get the latest occurrence of event_type so that we can find the difference between now() and also filter only those events which are having rowno 2 , i.e. consider only the last two events of event type and then filter only one where having the diff of latest and second latest

    |eval latest_diff=round((now()-latest_time)/60,0)
    |where latest_diff>diff

Find the difference between current time and latest available and compare it with the difference of latest & second latest

Hope this works for you

Re: How do I make an alert which triggers if an event's delay is unusually long?

woodcock — Wed, 17 Oct 2018 23:19:30 GMT

Schedule this search to run every Whatever minutes:

| gentimes start=10/16/18 end=10/17/18 increment=1h
| eval event_type = 1
| rename starttime AS _time

| rename COMMENT AS "Everything above generates sample events; everything below is your solution"

| table _time event_type
| dedup 2 event_type
| streamstats window=2 range(_time) AS delta BY event_type
| search delta>0
| eval delay = now() - _time
| eval threshold = 5 + delta
| where delay > threshold