Transactions With Variable start/end?

[dmoulais]

I have a log with many instances of the following ... these can happen in parallel. I was attempting to use a transaction to mine these via startswith="Received Query" and endswith="Completed Query".

<2018.10.29 10:02:37 639 -0400> Received Query on NE:38.120.48.29,{"targetClass":"nsd-service:/services/eline-sites/site","operator":"=","field":"classId","value":"nsd-service:/services/eline-sites/site"}, For BOTH Attributes
<2018.10.29 10:02:37 639 -0400> Received Query on NE:38.120.48.29,{"targetClass":"nsd-service:/services/eline-sites/site","operator":"=","field":"classId","value":"nsd-service:/services/eline-sites/site"}, For BOTH Attributes
<2018.10.29 10:02:37 639 -0400> Received Query on NE:38.120.48.29,{"targetClass":"nsd-service:/services/eline-sites/site","operator":"=","field":"classId","value":"nsd-service:/services/eline-sites/site"}, For BOTH Attributes
<2018.10.29 10:02:38 696 -0400> Completed Query: NE 38.120.48.29 Target Class nsd-service:/services/eline-sites/site
<2018.10.29 10:02:38 696 -0400> Completed Query: NE 38.120.48.29 Target Class nsd-service:/services/eline-sites/site
<2018.10.29 10:02:38 696 -0400> Completed Query: NE 38.120.48.29 Target Class nsd-service:/services/eline-sites/site

The problem is that since these can happen in parallel, I can mismatch start and end events. The way you know which ones match are based on the thread number ... 24 or 25 or 26 in this case. Is there a way to extract that dynamic number and use it to form the transaction ...

transaction startswith="Received Query" and endswith="Completed Query" ... and "contains" the dynamically generated number?

[kmaron]

is the thread number in your event? or can you pull it out with a regex? If you can you could do something like this.

| transaction ThreadNum  startswith="Received Query" endswith="CompletedQuery" maxevents=2

[dmoulais]

Yes this works thanks - I frequently forget you can use a field name as a transaction grouping 🙂

index=xxhost=vm4 source="/opt/nsp/mediation/log/xx.log" "xx-grpc-exec" AND ( "Received Query" OR "Completed Query" ) | rex "-exec[(?\d+)]" | transaction threadId startswith="Received Query" endswith="Completed Query" | chart values(duration) as "Query Time" by _time

[kmaron]

perfect! I'll convert it to an answer so you can accept it.