Splunk Search

Why does dedup not return any results?

Scan001
Explorer

Below is an example of a log file I'm trying to analyse (thousands of entries). I wish to remove duplicate entries based on the Acct-Session-Id. So I'm using dedup e.g.: source="file1" dedup Acct-Session-Id

What I get is; "No results found."

Is there something I'm missing? I have tried all suggestions on this forum.

Sun Jun  2 23:54:41 2014
    Packet-Type = Access-Request
    Acct-Session-Id = "6885EAB8-8056F22CA0AB-0000016600"
    Calling-Station-Id = "80-xx-xx-2xx-xx-AB"
    Called-Station-Id = "00-xx-xx-75-86-D0"
    Vendor-388-Attr-2 = 0xxxx475726f616d
    NAS-Port = 1
    NAS-Port-Type = Wireless-802.11
Tags (1)
0 Karma

chimell
Motivator

hi Scan001
Try search code with uniq command

 source="file1"  |table Acct-Session-Id| uniq
0 Karma

Scan001
Explorer

Thanks Chimell,

Unfortunately that returns all records and drops none of the duplicates.

0 Karma

gyslainlatsa
Motivator

hi,
we must put the pipe before using dedup because dedup is a command
dedup Removes the events which contain an identical combination of values for selected fields.
Also check if the field acc-session_id used by dedup appears in highlight the results.
because if acc-session_id is a field, it will not work.

check and let me know.

0 Karma

Scan001
Explorer

Hey,
Thanks for quick answer, I have tried it with and without the pipe. It does try and run when I use the pipe but returns zero results.

Any ideas?

0 Karma

Scan001
Explorer

Hey.

Okay I don't understand the second part of your answer. This may be the source of my problem. What do you mean
" if the field acc-session_id used by dedup appears in highlight the results. because if acc-session_id is a field....."

Apologise if this is a very basic question, I'm a newbe and I'm just getting the hang of the language..

0 Karma

gyslainlatsa
Motivator

I just ask to check if the Acct-Session-id field appears in the events and if multiple values
try this query: source="file1" |table Acct-Session-Id |dedup Acct-Session-Id an let me know if you have the results.

0 Karma

Scan001
Explorer

Yes, it is in every record. I tried your suggestion, but the duplicates are not filtered out, the complete set is returned.

Frustrating!

0 Karma

gyslainlatsa
Motivator

when you remove dedup, you have the results?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...