topic Re: Which version of dedup is better? in Splunk Search

Which version of dedup is better?

danielbb — Tue, 09 May 2023 18:35:58 GMT

A colleague of mine uses the following dedup version:

| strcat entity "-" IP "-" QID "-" Port "-" Tracking_Method "-" Last_Detected Key | dedup Key

And I grew up with

| dedup entity IP QID Port Tracking_Method Last_Detected

One caveat is Tracking_Method doesn't always exist. So which version is better?

Re: Which version of dedup is better?

goncalocoelho — Tue, 09 May 2023 20:36:11 GMT

Hi,

If Tracking_Method doesn't exist, I would write this:

(...) | stats count values(Tracking_Method) by entity IP QID Port Last_Detected

If you put it in the "by clause", it may not present all the desired results.

I believe that the stats offer a slight better performance than dedup. But you can test both options and check the job inspector for time and inspected events vs return events.

Re: Which version of dedup is better?

richgalloway — Tue, 09 May 2023 21:44:50 GMT

I've never seen anything conclusive about whether dedup or stats is faster. It may depend on other factors.

One significant difference, however, is stats is an aggregating command. That means the original events will be lost. Any field not mentioned in the command will be discarded. The output of the values function will be a multi-value field, which requires special handling later in the query. This is why I prefer dedup.

Re: Which version of dedup is better?

yuanliu — Wed, 10 May 2023 09:44:15 GMT

You can transform null into blank string before dedup, like this

| fillnull value="" Tracking_Method Port | dedup entity IP QID Port Tracking_Method Last_Detected

In theory, comparing a single field is less computation; however, strcat is not a simple task like fillnull. In my unscientific test, they perform about the same. (BTW, Port is likely to be null while Tracking_Method should always have value.)