Splunk Search

How to find time between similar events?

dl-it-serveradm
Engager

Hello,

We are looking to create a search that will return when two similar events occur within 1 second of each other.

Sample log search results:

2022-04-19 18:42:39,210 INFO [stdout] (default task-43) [core.service.RestService] ==============POST Send Family=============
2022-04-19 18:39:31,142 INFO [stdout] (default task-43) [core.service.RestService] ==============POST Send Family=============
2022-04-19 18:35:38,403 INFO [stdout] (default task-41) [core.service.RestService] ==============POST Send Family=============
2022-04-19 18:35:38,371 INFO [stdout] (default task-42) [core.service.RestService] ==============POST Send Family=============
2022-04-19 18:34:01,696 INFO [stdout] (default task-40) [core.service.RestService] ==============POST Send Family=============
2022-04-19 18:30:36,450 INFO [stdout] (default task-39) [core.service.RestService] ==============POST Send Family=============
2022-04-19 16:57:39,144 INFO [stdout] (default task-36) [core.service.RestService] ==============POST Send Family=============
2022-04-19 14:01:42,904 INFO [stdout] (default task-153) [core.service.RestService] ==============POST Send Family=============
2022-04-19 13:46:00,629 INFO [stdout] (default task-153) [core.service.RestService] ==============POST Send Family=============
2022-04-19 13:42:39,944 INFO [stdout] (default task-153) [core.service.RestService] ==============POST Send Family=============
2022-04-19 13:32:59,488 INFO [stdout] (default task-145) [core.service.RestService] ==============POST Send Family=============

 

We would like a query to be able to return results when events occur, like the following times, since they are so close together:

2022-04-19 18:35:38,403 INFO [stdout] (default task-41) [core.service.RestService] ==============POST Send Family=============
2022-04-19 18:35:38,371 INFO [stdout] (default task-42) [core.service.RestService] ==============POST Send Family=============

Is there a way we can generate a query that would find something like that?  

Thanks!

 

 

Labels (1)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

You need to use

streamstats window=1 current=f  values(_time) as prevtime

to find timestamp of previous occurrence of your event (you can add a "by" clause if you can distinguish different kinds of "similar" events). Then you can only filter the ones

 where _time - prevtime <1

The downside to this method is that you'll only get the second event of the two this way (or, if there are many subsequent quickly happening events - all of them except the first one).

You could also fiddle with the transaction command but it has its drawbacks and limitation.

View solution in original post

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You need to use

streamstats window=1 current=f  values(_time) as prevtime

to find timestamp of previous occurrence of your event (you can add a "by" clause if you can distinguish different kinds of "similar" events). Then you can only filter the ones

 where _time - prevtime <1

The downside to this method is that you'll only get the second event of the two this way (or, if there are many subsequent quickly happening events - all of them except the first one).

You could also fiddle with the transaction command but it has its drawbacks and limitation.

0 Karma

dl-it-serveradm
Engager

Hi, this is very close.  I'm having trouble calculating the "_time - prevtime".

The prevtime is calculating the following values:

<query> | streamstats window=1 current=f values(_time) as prevtime | table _raw prevtime

2022-04-19 18:39:31,142 INFO [stdout] (default task-43) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650408159.210
2022-04-19 18:35:38,403 INFO [stdout] (default task-41) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650407971.142
2022-04-19 18:35:38,371 INFO [stdout] (default task-42) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650407738.403
2022-04-19 18:34:01,696 INFO [stdout] (default task-40) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650407738.371
2022-04-19 18:30:36,450 INFO [stdout] (default task-39) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650407641.696
2022-04-19 16:57:39,144 INFO [stdout] (default task-36) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650407436.450
2022-04-19 14:01:42,904 INFO [stdout] (default task-153) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650401859.144
2022-04-19 13:46:00,629 INFO [stdout] (default task-153) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650391302.904
2022-04-19 13:42:39,944 INFO [stdout] (default task-153) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650390360.629
2022-04-19 13:32:59,488 INFO [stdout] (default task-145) [core.service.OpsManagerRestService] ==============POST Send Family============= 1650390159.944

When running this query, it still returns all events:

<query> | streamstats window=1 current=f values(_time) as prevtime | where _time-prevtime <1

 

Thanks again. 

 

0 Karma

dl-it-serveradm
Engager

Ignore prior post.  Was able to use this syntax:

 

<query> | streamstats window=1 current=f values(_time) as prevtime | eval "TimeDiff" = prevtime - _time | where TimeDiff<1

 

Thanks for your help!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...