Splunk Search

How to remove all identical events and keep the latest event for each different timestamp in a transaction search?

kkarthik2
New Member

We tried this search below:

index=test | eval dup=_raw | convert ctime(_time) as T1 |  transaction dup mvlist=t  maxspan=1s keepevicted=true | where eventcount>1

And these are sample events:

_time : 4/9/2015 12:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
_time : 4/9/2015 12:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
_time : 4/9/2015 12:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 13:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 13:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 14:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.
time : 4/9/2015 15:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.

I want latest events like the example below, remove the other events without using dedup command,
time : 4/9/2015 15:00:01 :-----host:foo1, source:foo2, source type:foo3, Date:foo4,status=foo5,Name=foo6.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Another strategy for this one, faster than dedup if the same fields are consistently present:

 index=test 
| stats  latest(_time) as _time latest(_raw) as _raw by host source sourcetype Date status Name 

...or ...

 index=test 
| stats  latest(_time) as _time by _raw

...but the second one means you need to re-extract the other fields.

0 Karma

devin_stonecyph
Explorer

Why do you want to avoid using dedup? Dedup is by far the easiest way of doing this. Just |dedup host, source, sourcetype, Date, Name | , and that's all you need to do. Add that before the transaction.

Dedup will leave you with the most recent result from each duplicate set.

From the docs: "Events returned by dedup will be the first event found in search order, which is the most recent for historical search"

0 Karma
Get Updates on the Splunk Community!

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...