topic Re: Alternate to dedup in Splunk Search

Alternate to dedup

rupesh26 — Tue, 28 Jan 2020 22:22:58 GMT

Hi,

I need to remove duplicates in my results, is there anyway to do this other than using dedup.
I used stats, eventstats still no luck

Re: Alternate to dedup

jscraig2006 — Tue, 28 Jan 2020 22:54:10 GMT

@ rupesh26 try a distinct count:
| stats dc(<your_feild>)

Re: Alternate to dedup

nick405060 — Tue, 28 Jan 2020 22:56:36 GMT

This counts distinct values it does not dedup.

Re: Alternate to dedup

rupesh26 — Tue, 28 Jan 2020 22:58:09 GMT

Yes, that's correct

Re: Alternate to dedup

nick405060 — Tue, 28 Jan 2020 23:00:32 GMT

So... this does not answer the question lol

Re: Alternate to dedup

nick405060 — Tue, 28 Jan 2020 23:05:12 GMT

stats count by your_field is faster than dedup if you don't want to keep other fields

Re: Alternate to dedup

rupesh26 — Tue, 28 Jan 2020 23:09:40 GMT

Thanks nick, by I want to keep other fields as well to add it to a dashboard.

Re: Alternate to dedup

jscraig2006 — Tue, 28 Jan 2020 23:20:23 GMT

Apologies! I should have read the question more carefully!

Re: Alternate to dedup

nick405060 — Tue, 28 Jan 2020 23:33:51 GMT

Terribly inelegant, but you could stats count by your_field and then join those results with the same search copied and pasted

<your_search> ... | table your_field b c | stats count by your_field | join type=left your_field [<your_search>] | table your_field b c

Re: Alternate to dedup

rupesh26 — Tue, 28 Jan 2020 23:37:19 GMT

Really appreciate it Nick , I will try these options.

Re: Alternate to dedup

nick405060 — Tue, 28 Jan 2020 23:46:19 GMT

Also for reference

https://answers.splunk.com/answers/789749/dedup-vs-stats-performance.html

(I am on the same page as you in that 99.9999% of the time I want to keep my other fields as well, which makes stats values absolutely useless in this "debate")