topic Re: Dedup removing all elements in Splunk Search

Dedup removing all elements

pedropiin — Mon, 10 Mar 2025 15:24:15 GMT

Hello everyone.

I'm dealing with a query that deals with certain "tickets" and "events", but some of them are duplicates, that's why it runs a dedup command. But there seems to be something else happening.

The query is of the form:

Listing all the events, I'm able to see that the, basically, the main duplicate events are the ones that were null and were filled with "[empty]". But, for some reason, some of the events disappear with dedup.

In theory, dedup should remove all duplicates and maintain one, representing all of its "copies". And that happens for some "names", but not for all. During the same query, I deal with events of the category "name1" and events of the category "name2". All of theirs instances are "[empty]", and running dedup removes all instances of "name1" and maintains one of "name2", when it should maintain one of both.

Why is that happening?

Each instance is of the form
" processTime | arrivalTime | name | tickets | count"

Re: Dedup removing all elements

richgalloway — Mon, 10 Mar 2025 15:53:18 GMT

The dedup command keeps the first event it finds for each unique value of the field(s) specified in its arguments ("tickets" in this case). The values of other fields are ignored. Depending on the sequence of events, it's entirely possible for each ticket value to come first from name1 and be retained and other names will be discarded.

If you need to dedup on both tickets and name then use dedup tickets name in the query.

Re: Dedup removing all elements

gcusello — Mon, 10 Mar 2025 16:03:21 GMT

Hi @pedropiin ,

the stats command automatically dedups values, so you don't need to use the dedup command before the stats command.

Ciao.

Giuseppe