Can we include OR/AND operator in a transaction

[amunag439]

I have the following log sets, one for success case and one for the failure case

Success:
id=11111 msg=Begin process...
id=11111 msg=check
id=11111 msg=Success...

failure:
id=22222 msg=Begin process...
id=22222 msg=check
id=22222 msg=Fail...

Here I want to check the time between the events using the transaction.

host=* sourcetype=** source="*/example.log" "Begin process*" OR "Success*"
  | transaction traceId startswith="Begin process" endswith="Success" 
  | table traceId duration _time

Above query will give me the transactions of a success case only.
Can we use AND Operator in the endswith so that I can check the duration between events irrespective of it being a success or failure?

[woodcock]

There is only one use case where the use of transaction is merited but this command scales so poorly that I am not even going to mention it. Stop using transaction; try this:

host=* sourcetype=** source="*/example.log" "Begin process*" OR "Success*" OR "Failure*"
| streamstats count(eval(searchmatch("Success* OR Failure*"))) AS sessionID BY traceId
| stats range(_time) AS duration list(_raw) AS events min(_time) AS _time BY traceId sessionID

[niketn · Accepted solution]

@amunag439 I would recommend using stats instead of transaction for this scenario

 <yourCurrentCodeToGetTimeMsgAndID>
|  stats list(msg) as msg_all values(msg) as msg_unique min(_time) as _time max(_time) as latest_time by id
|  eval duration=latest_time-_time
|  fields - latest_time
|  eval startswith=mvindex(msg_all,0),endswith=mvindex(msg_all,mvcount(msg_all)-1)
|  fields - msg_all msg_unique
|  search startswith="Begin Process" endswith="*"

Following is a run anywhere example based on sample data and details provided. It looks only for startswith condition. Endwith any value is accepted (you can explicitly set to Success and Fail as well if there are only two final messages in your data).

|  makeresults
|  eval _time=relative_time(now(),"-1d")
|  eval data="id=11111 msg=Begin process;id=11111 msg=check;id=11111 msg=Success;id=22222 msg=Begin process;id=22222 msg=check;id=22222 msg=Fail"
|  makemv data delim=";"
|  mvexpand data
|  rename data as _raw
|  KV
|  eval msg=replace(msg,"Begin","Begin Process")
|  eval delta_duration=random()
|  eval delta_duration=substr(delta_duration,1,3)
|  accum delta_duration
|  eval _time=_time+delta_duration
|  fields - _raw delta_duration
|  stats list(msg) as msg_all values(msg) as msg_unique min(_time) as _time max(_time) as latest_time by id
|  eval duration=latest_time-_time
|  fields - latest_time
|  eval startswith=mvindex(msg_all,0),endswith=mvindex(msg_all,mvcount(msg_all)-1)
|  fields - msg_all msg_unique
|  search startswith="Begin Process" endswith="*"

[amunag439]

@niketnilay in the failure case what if there are few more logs? if Fail is not a final log, how do we approach that?

[niketn]

@amunag439 the query that I have provided is similar to what transaction will do when there is starts with condition but ends with can be anything.

If you want other more specific conditions you will have to play with msg_unique which has all distinct values of msg field present for specific ids. Final 2 pipes need to change as follows.

  |  fields - msg_all
  |  search startswith="Begin Process" endswith="*"

msg_unique=Success AND msg_unique=Fail.