Transaction | Include all events, even if they occurred outside of search time range?

[bmohammadi]

Dear Community,

Lets say I was running a search for an hour period from 10:00 until 11:00 and we had a particular transaction that consisted of 2 or more events - the first occurring at 09:59 and the last at 10:01. 

Using the default Transaction command any events which occurred before 10:00 would not be included and we would therefore not be viewing the whole transaction. Likewise, if a transaction started at 10:59 and didn't end until 11:01, any events which occurred after 11:00 would be dropped.

Is there any way to include all events related to transactions which started or ended during the specified search time range?

Conversely, If this is not possible it would be helpful to drop any transactions which did not start and end within the time range - is there any way to achieve this?

Kind regards,

Ben

[PickleRick]

1. No. Your search range determines the set of events which you will be processing (statsing, combining into transactions and so on). There is no way to "pull" additional events bar some ugly tricks like spawning a subsearch with map. But this is not something that I would recommend even considering.

2. In general, transaction is not a command which should be used lightly as it is relatively resource-intensive and has its limitations. If you can use fancy grouping and stats, do so.

3. If you use the transaction command, the _time of the combined transaction event is that of the earliest "source" event from this transaction. So you can filter on that field after combining your events. Alternatively, if you use stats instead, you can calculate earliest(_time) from your events group and filter on that.

[bmohammadi]

Hi PickleRick,

Thank you very much for your very prompt response.

1. Then I wont consider it
   
   
2. As far as I am aware it is impossible for me to achieve what I need without using transaction. I need to categorise the 'result' of each transaction (using various if statements) based on a combination of different fields extracted from different event types that make up each transaction, then generate a timechart by count of result category. I do not know how to do this using stats.
   
   
3. So theoretically I could drop transactions that end within, but start outside of, a specific time range by setting the search range to earlier than I need, then filtering transactions where _time is less than the desired start time?
   
   Can you offer any advise on how to do this? Obviously I would need to filter on the _time filed after the transaction statement - I have tried various methods but none have worked.

Kind regards,

Ben

[PickleRick]

1. Good 🙂

2. Can't help you without a proper sample of events and description of desired outcome but it is possible that transaction is the only solution. It sometimes happens (the command is there for a reason after all ;-)). It's just that performance-wise it's often better to find another way for your search. Especially if it's gonna be invoked often and over a big set of data. If it's a one-off, not worth the effort probably.

3. For example something like that:

index=whatever other=conditions earliest=-2d@d latest=-1d@d
| <your evals, transactions and whatnot>
| where _time<relative_time(now(),"-1d@d-6h")

The last "where" command limits your results only to those that started not later than 6 hours before your "latest" parameter of the search. Which means that you're searching within a day - from -2d@d till -1d@d but later filter out 6 latest hours from this range

[ITWhisperer]

Additionally, if you are going to use transaction, there are the keeporphans and keepevicted options for dealing with incomplete transactions

[bmohammadi]

Hi ITWhisperer,

Thank you for taking the time to respond.

I tried various combinations of keeporphans and keepevicted  but it didn't have any effect on transactions that ran over the search time range