How to extract list of event ids in shorter time range and search for all correlated events in larger time range?

[fredv44]

Hello.

Given these logs:

2022-03-16 16:08:43.991 traceId="7890" svc="Service1" duration=132
2022-03-16 16:10:43.279 traceId="1234" svc="Service1" duration=132
2022-03-16 16:38:43.281 traceId="5678" svc="Service3" duration=219
2022-03-16 16:43:43.284 traceId="1234" svc="Service2" duration=320
2022-03-16 17:03:44.010 traceId="1234" svc="Service2" duration=1023
2022-03-16 17:04:44.299 traceId="5678" svc="Service3" duration=822
2022-03-16 17:19:44.579 traceId="5678" svc="Service2" duration=340
2022-03-16 17:32:44.928 traceId="1234" svc="Service1" duration=543

I would like in a single search to:

• extract all traceIds that happened between 17:00 and 17:05
• search for the captured traceIds in larger range (say between 16:00 and 18:00)

Is that possible? Thank you!

[fredv44]

Thanks @gcusello 

In fact the reason why I want to extract in a small range first is to narrow down my search (a sort of event sampling). There are millions of traceIds in this 2-hour range and search with traceId=* doesn't go through (timeout). I am looking to extract a subset of traceIds in this small range first so that I can look for all events that happened for those in this 2-hour range (hoping that Splunk will be able to efficiently use those traceIds to filter out millions of rows and not time out).

Of course, a traceId can span over multiple minutes, hence a 5-minute range will be too short to capture what happened with those captured traceIds.

I am intending to run this query at different dates/times in the past.

I hope it makes sense 🙂

[gcusello]

Hi @fredv44,

surely, my answer isn't a final solution, but an approach that you have to customize for your present and future needs.

If this solution answers to your request, please accept it for the other people of Community, otherwise, please, tell me how can I help you more.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

[gcusello · Accepted solution]

Hi @fredv44,

let me understand, you want to find the TraceID present between 17.00 and 17.05 and see if they are also present between 16.00 and 18.00?

If this is your need you could try something like this:

I suppose to run your seach at 18.00:

index=your_index traceId=* earliest=-120m@m latest=@m
| eval 
     little_period_start=relative_time(now(),"-60m@m"),
     little_period_end=relative_time(now(),"-55m@m")
| eval period=if(_time>little_period_start AND _time<=little_period_end,"inside","outside")
| stats dc(period) AS dc_period values(period) AS period BY traceId

In this way you can configure all the conditions you want: only inside, only outside, in both periods using a search command at the end of the search.

Ciao.

Giuseppe

[gcusello]

Hi @fredv44,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)