Splunk Search
Highlighted

dedup and unique difference

Motivator

Could you please explain the difference between dedup and unique

Tags (2)
0 Karma
Highlighted

Re: dedup and unique difference

Splunk Employee
Splunk Employee

The uniq command removes duplicates if the whole event or row of a table are the same. It takes no fields or options as everything is checked. It is an ideal command if you have duplicate data.
See docs on uniq for more detail.

The dedup command looks only at the fields you tell it to. So if I say "| dedup host", it only looks at the host field and keeps the first from each host. You can specify multiple fields and has options like consecutive (only remove events with duplicate combinations of values that are in consecutive rows.) or keepempty (also keep events that do not have the requested field).
See docs on dedup for more detail

View solution in original post

Highlighted

Re: dedup and unique difference

Legend

Agree, please use Splunk Documentation as your first point of research, or be more specific which what is your use case or reason for the question.




| eval message="Happy Splunking!!!"


0 Karma
Highlighted

Re: dedup and unique difference

Legend

@logloganathan, I see that you have down voted my comment. Down voting should only be reserved for suggestions/solutions that could be potentially harmful for a Splunk environment or goes completely against known best practices.

Simply commenting with more information about what didn't work and what you've tried (or whatever other info may be relevant) would suffice to help you troubleshoot further.

Refer to community guidelines (ironically again on Splunk Docs :)): https://docs.splunk.com/Documentation/Splunkbase/splunkbase/Answers/Splunkcommunityguidelines

I am curious to know as to how request to research on own before asking question is harmful for you/your environment. Please clarify!!!




| eval message="Happy Splunking!!!"


0 Karma
Highlighted

Re: dedup and unique difference

Esteemed Legend

The uniq command removes any search result if that result is an exact duplicate so the events must be resorted to use it. I have NEVER had any occasion to use this command. Ever. The dedup command is MUCH more flexible. Unlike uniq It can be map-reduced, it can trim to a certain size (defaults to 1) and can apply to any number of fields at the same time.