Splunk Search

Alternative to dedup

greggz
Communicator

I'm sorting by time cause I want the latest time for every distinct host. Im doing this and it works. But dedup is fairly heavy.. I've tried searching for alternatives but to no avail. Can you guys point me in the right direction ? thanks

Be mindfull that I have a lot more fields in the stats pipe

 | stats
       count(duration) as request_count 
        sum(duration) as total_duration
        sum(exc_thrown) as error_count
        .....  by host _time | sort - _time  | dedup host

I currently have uptaded to | dedup host stortby -_time instead of | sort - _time | dedup host. Don't know which is faster

Tags (2)
0 Karma

mayurr98
Super Champion

you can check execution time in jobs>inspect job. Using dedup on larger dataset can be expensive. There are cases where you can replace dedup by using a stats latest(...

  | stats
latest(_time) as latest_time        
count(duration) as request_count 
         sum(duration) as total_duration
         sum(exc_thrown) as error_count
         .....  by host | sort -  latest_time  | eval latest_time=strftime(latest_time,"%d-%m-%Y %H:%M:%S")

let me know if this helps!

0 Karma

greggz
Communicator

No it's not helping, I sort by host and _time because I need one host at the specific time for all distinct hosts. This code pulling in values from other times to the same host giving false values

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...