topic Alternative to dedup in Splunk Search

Alternative to dedup

greggz — Wed, 24 Jan 2018 10:20:11 GMT

I'm sorting by time cause I want the latest time for every distinct host. Im doing this and it works. But dedup is fairly heavy.. I've tried searching for alternatives but to no avail. Can you guys point me in the right direction ? thanks

Be mindfull that I have a lot more fields in the stats pipe

 | stats
       count(duration) as request_count 
        sum(duration) as total_duration
        sum(exc_thrown) as error_count
        .....  by host _time | sort - _time  | dedup host

I currently have uptaded to | dedup host stortby -_time instead of | sort - _time | dedup host. Don't know which is faster

Re: Alternative to dedup

mayurr98 — Wed, 24 Jan 2018 10:38:22 GMT

you can check execution time in jobs>inspect job. Using dedup on larger dataset can be expensive. There are cases where you can replace dedup by using a stats latest(...

  | stats
latest(_time) as latest_time        
count(duration) as request_count 
         sum(duration) as total_duration
         sum(exc_thrown) as error_count
         .....  by host | sort -  latest_time  | eval latest_time=strftime(latest_time,"%d-%m-%Y %H:%M:%S")

let me know if this helps!

Re: Alternative to dedup

greggz — Wed, 24 Jan 2018 10:59:36 GMT

No it's not helping, I sort by host and _time because I need one host at the specific time for all distinct hosts. This code pulling in values from other times to the same host giving false values