Splunk Search

Dedup duplicates

Motivator

Hi,

what is the easiest way to filter out event duplicates without adding every field in the dedup command?
Is
| dedup _raw

the correct approach?

BR

Tags (2)
1 Solution

Legend

dedup _raw should work just fine, yes.

View solution in original post

Motivator

I've got two additional questions regarding this topic:

  1. How can I search for the count of events that have duplicates?
  2. How can I search for the total number of duplicates?

BR

Heinz

0 Karma

Motivator

Unfortunately I don't have an unique identifier for each event like your proposed session_id

0 Karma

Communicator

You can count duplicated event by using the "transaction" command. And then count the events by using "eventcount"

eg.:

eventtype="*" | transaction session_id | Where eventcount>1 | stats count by eventcount

to find out how many duplicates occured

or:

eventtype="*" | transaction session_id | Where eventcount>1 | stats count(eventcount)

to count how many different duplicated events occured

or ...

0 Karma

Legend

dedup _raw should work just fine, yes.

View solution in original post

Motivator

great, thanks

0 Karma

Explorer

When I try and enter the "|dedup _raw" command at the end of my search parameter I end up with no matches but when I take it off the end I end up with thousands. I can see that they are duplicates(same IP address, name, and port) but it still doesn't work. any suggestions?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!