How to Get Surrounding Logs by Thread ID Efficiently (and Accurately)

[hoyomi]

What I want to achieve is to extract surrounding log lines by thread ID, which is found on the line where the keyword is seen.

A sample snippet of my log looks like

2018-01-01 01:00:01,000 [Module thread#2] info - action starts
2018-01-01 01:00:01,010 [Module thread#3] info - unrelated log
2018-01-01 01:00:01,011 [Module thread#2] info - look for user some.one@email.com
2018-01-01 01:00:01,012 [Module thread#4] info - other unrelated log
2018-01-01 01:00:01,030 [Module thread#2] info - found record in DB
2018-01-01 01:00:01,034 [Module thread#2] info - checking user permission
2018-01-01 01:00:01,034 [Module thread#3] info - unrelated log
2018-01-01 01:00:01,040 [Module thread#2] error - user has no permission for this action
...
2018-01-01 01:00:15,123 [Module thread#2] info - another action starts reusing thread ID #2
...

When I search for some.one@email.com and get _time (2018-01-01 01:00:01,011) and thread ID (Module thread#2) using rex, I want to extract everything associated with thread#2 within a range of +/- 30-second window. But I don't want to extract the last line in the example above, which is part of another set of logs for a different action. So expected result should only contain line 1, 3, 5, 6 and 8.

One way I know is to get _time and thread ID and use them in a subsequent map search on the same log again with earliest=_time-30 and latest=_time+30 (roughly like this). But it would be tricky to get rid of the last line which falls into the window but is unrelated.

When googling around, someone mentioned transaction. So I tried to use transaction with maxspan=30s maxpause=5s. But it seems only returning lines with the keyword.

Can someone please suggest an efficient and accurate way to achieve this?

[Sukisen1981]

The last line that you are mentioning, can it be identified with something unique?
Is the row number of the last line always a multiple of 10 or something like that
OR
is text like 'another action starts' a common and unique identifier for the last line?

What I am trying to say is this - what if we approach this issue from the other side? Meaning, we extract ALL the matching rex, which is most probably multi valued, then expand the multi valued rex AND filter out the unwanted line?
Possible?

[hoyomi]

Hi @Sukisen1981

Unfortunately, there is no easy way because the thread can be used by some other module which follows a completely different format of logging.

[Sukisen1981]

well then how do you know what the 'last line' is?