Does using dedup improve performance? And where to place it in search?

[kamryn]

I just finished the Splunk Fundamentals 1 course and in one of the videos they said:

For best performance we place dedup as early in the search as possible.

However I was just reviewing Splunk documentation and found this entry:

Which of these two suggestions is the most correct? Also, does using dedup in searches have any benefit to performance?

[koshyk · Accepted solution]

The commands like stats, dedup ( I won't include sort with this), is quite useful and doing it early makes the total dataset much less

Let's take an example
1) index=windows | eval newUser=user| stats count by newUser
vs
2) index=windows| stats count by user| rename user as newUser

in (1), you are doing an evaluation/renaming on every single event. But in (2), you are aggregating and then doing rename, which may be only for handful of data

the key part is stats and dedup reduces the data/rows to very minimal than doing it on all data

Please do analysis yourself and see how the speed/performance difference is using your dataset.