Splunk Search

How do I make an alert which triggers if an event's delay is unusually long?

matthewg
Explorer

I have multiple events such as below:

Key points here:

  • New values of event_type may be added randomly and the schedule may be randomly changed for different event types.

    • I just want an alert for if an event gets missed or delayed more than x (seconds/minutes/hours) longer than between the previous events
    • Then I want it to see the new normal for that event type.
    • Specifically: if the difference in time between now and the last event is more than x (seconds/minutes/hours) greater than the difference between the last event and the previous one, I want to trigger an alert.

    8:03am event_type=1
    8:05am event_type=2
    8:15am event_type=2
    8:25am event_type=2
    8:33am event_type=1
    9:15am event_type=2
    10:05am event_type=2
    Example: The search should see that event type=1 happened at 8:03 and at 8:33 (difference 30 min) so the next event_type=1 should be at 9:03. If x is 5 min., then at 9:08, an alert should be triggered if event_type=1 is not seen.

Similarly, event type=2 happened at 8:15 and 8:25 (difference 10 min) so if x is 5 min then at 8:40 an alert should be triggered for a missing event_type=2.

In this example someone changed the schedule for event_type=2 (to 50 minutes), so after seeing an event at 9:15, it should expect the next one at 10:05 since the difference between the last two was 50 minutes.

What I tried:

event_type="*" | streamstats current=f window=2 earliest(_time) as time_2 latest(_time) as time_1 by event_type| eval time_0_diff = time_1 - _time  | eval time_1_diff=time_2 - time_1 | eval time_variance=time_0_diff - time_1_diff |

This only works if I have an event though and I am specifically looking for when an event is missing (or delayed).

0 Karma
1 Solution

renjith_nair
SplunkTrust
SplunkTrust

@matthewg,

Try

event_type="*"
|sort event_type,- _time 
|streamstats last(_time) as prev_time by event_type current=f window=1|eval diff=round((prev_time-_time)/60,0)
|streamstats count as rowno,latest(_time) as latest_time by event_type|where rowno==2 
|eval latest_diff=round((now()-latest_time)/60,0)
|where latest_diff>diff

    |sort event_type,- _time 
    |streamstats last(_time) as prev_time by event_type current=f window=1|eval diff=round((prev_time-_time)/60,0)

Sorted event_type in ascending and _time in descending so that the similar events are adjacent and sorted in the order of _time(latest first). Then calculate the difference between the current latest and second latest in minutes

    |streamstats count as rowno,latest(_time) as latest_time by event_type|where rowno==2 

Get the latest occurrence of event_type so that we can find the difference between now() and also filter only those events which are having rowno 2 , i.e. consider only the last two events of event type and then filter only one where having the diff of latest and second latest

    |eval latest_diff=round((now()-latest_time)/60,0)
    |where latest_diff>diff

Find the difference between current time and latest available and compare it with the difference of latest & second latest

Hope this works for you

Happy Splunking!

View solution in original post

woodcock
Esteemed Legend

Schedule this search to run every Whatever minutes:

| gentimes start=10/16/18 end=10/17/18 increment=1h
| eval event_type = 1
| rename starttime AS _time

| rename COMMENT AS "Everything above generates sample events; everything below is your solution"

| table _time event_type
| dedup 2 event_type
| streamstats window=2 range(_time) AS delta BY event_type
| search delta>0
| eval delay = now() - _time
| eval threshold = 5 + delta
| where delay > threshold

renjith_nair
SplunkTrust
SplunkTrust

@matthewg,

Try

event_type="*"
|sort event_type,- _time 
|streamstats last(_time) as prev_time by event_type current=f window=1|eval diff=round((prev_time-_time)/60,0)
|streamstats count as rowno,latest(_time) as latest_time by event_type|where rowno==2 
|eval latest_diff=round((now()-latest_time)/60,0)
|where latest_diff>diff

    |sort event_type,- _time 
    |streamstats last(_time) as prev_time by event_type current=f window=1|eval diff=round((prev_time-_time)/60,0)

Sorted event_type in ascending and _time in descending so that the similar events are adjacent and sorted in the order of _time(latest first). Then calculate the difference between the current latest and second latest in minutes

    |streamstats count as rowno,latest(_time) as latest_time by event_type|where rowno==2 

Get the latest occurrence of event_type so that we can find the difference between now() and also filter only those events which are having rowno 2 , i.e. consider only the last two events of event type and then filter only one where having the diff of latest and second latest

    |eval latest_diff=round((now()-latest_time)/60,0)
    |where latest_diff>diff

Find the difference between current time and latest available and compare it with the difference of latest & second latest

Hope this works for you

Happy Splunking!
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...