Monitoring Splunk

Is there a performance impact by using Dedup command in SPL Queries?

uhkc777
Explorer

Hi,

I'm using dedup command in almost all my search queries. Does it have any impact on performance? If yes, what's the alternative for that?

Thanks,

0 Karma

pjvarjani
Path Finder

Dedup is absolutely ok with larger dataset also for your requirements. Since you want to do some logic on top of dedup, stats dc() and Head commands are out of picture here. Try to write two different queries that give same results but with different approaches given below and check in job inspect which query is faster.

  1. base query | dedup | your logic
  2. base query | stats latest.... by field(u want to dedup)

Thanks,
Pankaj

0 Karma

somesoni2
Revered Legend

Using dedup on larger dataset can be expensive. There are cases where you can replace dedup by using a stats latest(... OR subsearch as filters or something else. Whether dedup can be replaces OR not and if yes, then with what will depend upon your query requirements. Could you give some sample search on how the dedup is being used?

0 Karma

uhkc777
Explorer

index=test |dedup od,line|timechart span=1d count(od) as total|stats avg(total)

0 Karma

felipecerda
Path Finder

did you try this?:

index=test |dedup line|timechart span=1d dc(od) as total|stats avg(total)

0 Karma

uhkc777
Explorer

dedup line won't work in our scenario. I need to filter the events where od and line are same in the events.

0 Karma

somesoni2
Revered Legend

Can you compare your dedup results (and performance) with following query?

index=test | eval temp=od."#".line| timechart span=1d dc(temp) as total | stats avg(total)
0 Karma

uhkc777
Explorer

I think it's gonna work out. Thank you. I appreciate your support.

Is there any way to contact you through e-mail or phone?

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

uhkc777 - Did the search query provided by somesoni2 help provide a working solution to your question? Please let me know when you can so that it can be converted to an answer. Thanks!

0 Karma

felipecerda
Path Finder

It's better to use dc(your_field) whenever you can. I once asked what was the difference to a Splunk Instructor and he said that dc was faster than dedup.

0 Karma

uhkc777
Explorer

I don't want the count. I need to write some logic on top of that dedup command

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!