How to remove all identical events and keep the latest event for each different timestamp in a transaction search?

[kkarthik2]

We tried this search below:

index=test | eval dup=_raw | convert ctime(_time) as T1 |  transaction dup mvlist=t  maxspan=1s keepevicted=true | where eventcount>1

And these are sample events:

_time : 4/9/2015 12:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
_time : 4/9/2015 12:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
_time : 4/9/2015 12:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 13:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 13:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 14:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 15:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.

I want latest events like the example below, remove the other events without using dedup command,
time : 4/9/2015 15:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.

[DalJeanis]

Another strategy for this one, faster than dedup if the same fields are consistently present:

 index=test 
| stats  latest(_time) as _time latest(_raw) as _raw by host source sourcetype Date status Name 

...or ...

 index=test 
| stats  latest(_time) as _time by _raw

...but the second one means you need to re-extract the other fields.

[devin_stonecyph]

Why do you want to avoid using dedup? Dedup is by far the easiest way of doing this. Just |dedup host, source, sourcetype, Date, Name | , and that's all you need to do. Add that before the transaction.

Dedup will leave you with the most recent result from each duplicate set.

From the docs: "Events returned by dedup will be the first event found in search order, which is the most recent for historical search"