topic Why does dedup not return any results? in Splunk Search

Why does dedup not return any results?

Scan001 — Mon, 08 Jun 2015 15:20:38 GMT

Below is an example of a log file I'm trying to analyse (thousands of entries). I wish to remove duplicate entries based on the Acct-Session-Id. So I'm using dedup e.g.: source="file1" dedup Acct-Session-Id

What I get is; "No results found."

Is there something I'm missing? I have tried all suggestions on this forum.

Sun Jun  2 23:54:41 2014
    Packet-Type = Access-Request
    Acct-Session-Id = "6885EAB8-8056F22CA0AB-0000016600"
    Calling-Station-Id = "80-xx-xx-2xx-xx-AB"
    Called-Station-Id = "00-xx-xx-75-86-D0"
    Vendor-388-Attr-2 = 0xxxx475726f616d
    NAS-Port = 1
    NAS-Port-Type = Wireless-802.11

Re: Why does dedup not return any results?

gyslainlatsa — Mon, 28 Sep 2020 20:11:35 GMT

hi,
we must put the pipe before using dedup because dedup is a command
dedup Removes the events which contain an identical combination of values for selected fields.
Also check if the field acc-session_id used by dedup appears in highlight the results.
because if acc-session_id is a field, it will not work.

check and let me know.

Re: Why does dedup not return any results?

Scan001 — Mon, 08 Jun 2015 15:27:36 GMT

Hey,
Thanks for quick answer, I have tried it with and without the pipe. It does try and run when I use the pipe but returns zero results.

Any ideas?

Re: Why does dedup not return any results?

gyslainlatsa — Mon, 08 Jun 2015 15:36:13 GMT

when you remove dedup, you have the results?

Re: Why does dedup not return any results?

Scan001 — Mon, 28 Sep 2020 20:14:25 GMT

Hey.

Okay I don't understand the second part of your answer. This may be the source of my problem. What do you mean
" if the field acc-session_id used by dedup appears in highlight the results. because if acc-session_id is a field....."

Apologise if this is a very basic question, I'm a newbe and I'm just getting the hang of the language..

Re: Why does dedup not return any results?

chimell — Mon, 08 Jun 2015 16:23:20 GMT

hi Scan001
Try search code with uniq command

 source="file1"  |table Acct-Session-Id| uniq

Re: Why does dedup not return any results?

Scan001 — Mon, 08 Jun 2015 16:34:12 GMT

Thanks Chimell,

Unfortunately that returns all records and drops none of the duplicates.

Re: Why does dedup not return any results?

gyslainlatsa — Mon, 08 Jun 2015 16:34:37 GMT

I just ask to check if the Acct-Session-id field appears in the events and if multiple values
try this query: source="file1" |table Acct-Session-Id |dedup Acct-Session-Id an let me know if you have the results.

Re: Why does dedup not return any results?

Scan001 — Mon, 08 Jun 2015 22:40:05 GMT

Yes, it is in every record. I tried your suggestion, but the duplicates are not filtered out, the complete set is returned.

Frustrating!