How do I search for all events in the same "request" as a particular log line

[illuminatedaxis]

My application is a backend web service. All events in a request contain the same value for a "req_id" field.

I have a use-case, where I want to look at all the events that occurred for requests, only when a particular log line is present.

My first query would be this -

index="myapp" AND "some log line" | rex field=_raw "req_id=(?<req_id>[a-zA-Z0-9]*)"

And then my second query would be -

index="myapp" AND "$req_id" | transaction req_id

where the $req_id would be fed in by the first query.

How do I join these two queries? 

Thanks in advance!

[bowesmana]

P_vandereerden's reply is a good starting point, but there are two things to consider

1. The use of a subsearch to constrain an outer search may not perform well if there are a large number of requests ids with that log line. If you are expecting a large number of hits for "log_line" then you may need to consider a different approach.

2. The use of transaction has limitations and although it has use cases, it's options should be understood in relation to your data set, particularly when your data set is large.

Very often the stats command can be used to achieve the same thing as transaction without the limitations, so it very much depends on what you want to do with the resultant grouped data. For example this is generally a simple replacement for transaction

| stats values(_raw) as _raw range(_time) as duration count by requestId

which will give you the raw events, the duration from first to last and the number of events for any given request id.

[P_vandereerden · Accepted solution]

Use the first search as a subsearch:

index=myapp [ search index=myapp "some log line" | rex field=_raw "req_id=(?<req_id>[a-zA-Z0-9]*)" | table req_id ] | transaction req_id