Splunk Search

How to extract list of event ids in shorter time range and search for all correlated events in larger time range?

fredv44
Explorer

Hello.

Given these logs:

2022-03-16 16:08:43.991 traceId="7890" svc="Service1" duration=132
2022-03-16 16:10:43.279 traceId="1234" svc="Service1" duration=132

2022-03-16 16:38:43.281 traceId="5678" svc="Service3" duration=219
2022-03-16 16:43:43.284 traceId="1234" svc="Service2" duration=320
2022-03-16 17:03:44.010 traceId="1234" svc="Service2" duration=1023
2022-03-16 17:04:44.299 traceId="5678" svc="Service3" duration=822
2022-03-16 17:19:44.579 traceId="5678" svc="Service2" duration=340
2022-03-16 17:32:44.928 traceId="1234" svc="Service1" duration=543

I would like in a single search to:

  • extract all traceIds that happened between 17:00 and 17:05
  • search for the captured traceIds in larger range (say between 16:00 and 18:00)

Is that possible? Thank you!

Labels (2)
0 Karma
1 Solution

gcusello
Legend

Hi @fredv44,

let me understand, you want to find the TraceID present between 17.00 and 17.05 and see if they are also present between 16.00 and 18.00?

If this is your need you could try something like this:

I suppose to run your seach at 18.00:

index=your_index traceId=* earliest=-120m@m latest=@m
| eval 
     little_period_start=relative_time(now(),"-60m@m"),
     little_period_end=relative_time(now(),"-55m@m")
| eval period=if(_time>little_period_start AND _time<=little_period_end,"inside","outside")
| stats dc(period) AS dc_period values(period) AS period BY traceId

In this way you can configure all the conditions you want: only inside, only outside, in both periods using a search command at the end of the search.

Ciao.

Giuseppe

View solution in original post

fredv44
Explorer

Thanks @gcusello 

In fact the reason why I want to extract in a small range first is to narrow down my search (a sort of event sampling). There are millions of traceIds in this 2-hour range and search with traceId=* doesn't go through (timeout). I am looking to extract a subset of traceIds in this small range first so that I can look for all events that happened for those in this 2-hour range (hoping that Splunk will be able to efficiently use those traceIds to filter out millions of rows and not time out).

Of course, a traceId can span over multiple minutes, hence a 5-minute range will be too short to capture what happened with those captured traceIds.

I am intending to run this query at different dates/times in the past.

I hope it makes sense 🙂

0 Karma

gcusello
Legend

Hi @fredv44,

surely, my answer isn't a final solution, but an approach that you have to customize for your present and future needs.

If this solution answers to your request, please accept it for the other people of Community, otherwise, please, tell me how can I help you more.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

gcusello
Legend

Hi @fredv44,

let me understand, you want to find the TraceID present between 17.00 and 17.05 and see if they are also present between 16.00 and 18.00?

If this is your need you could try something like this:

I suppose to run your seach at 18.00:

index=your_index traceId=* earliest=-120m@m latest=@m
| eval 
     little_period_start=relative_time(now(),"-60m@m"),
     little_period_end=relative_time(now(),"-55m@m")
| eval period=if(_time>little_period_start AND _time<=little_period_end,"inside","outside")
| stats dc(period) AS dc_period values(period) AS period BY traceId

In this way you can configure all the conditions you want: only inside, only outside, in both periods using a search command at the end of the search.

Ciao.

Giuseppe

gcusello
Legend

Hi @fredv44,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)

0 Karma
Get Updates on the Splunk Community!

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Want a chance to win $500 to the Splunk shop? Take our IT Incident Management Survey!

  Top Trends & Best Practices in Incident ManagementSplunk is partnering up with Constellation Research to ...