Dedup duplicates

[HeinzWaescher]

Hi,

what is the easiest way to filter out event duplicates without adding every field in the dedup command?
Is
| dedup _raw

the correct approach?

BR

[HeinzWaescher]

I've got two additional questions regarding this topic:

1. How can I search for the count of events that have duplicates?
2. How can I search for the total number of duplicates?

BR

Heinz

[HeinzWaescher]

Unfortunately I don't have an unique identifier for each event like your proposed session_id

[Rocket66]

You can count duplicated event by using the "transaction" command. And then count the events by using "eventcount"

eg.:

eventtype="*" | transaction session_id | Where eventcount>1 | stats count by eventcount

to find out how many duplicates occured

or:

eventtype="*" | transaction session_id | Where eventcount>1 | stats count(eventcount)

to count how many different duplicated events occured

or ...

[Ayn · Accepted solution]

dedup _raw should work just fine, yes.

[HeinzWaescher]

great, thanks

[ITUser1]

When I try and enter the "|dedup _raw" command at the end of my search parameter I end up with no matches but when I take it off the end I end up with thousands. I can see that they are duplicates(same IP address, name, and port) but it still doesn't work. any suggestions?