topic Re: how can I use dedup command using many fields?? in Splunk Search

how can I use dedup command using many fields??

pacifikn — Thu, 19 Dec 2019 18:40:39 GMT

Greetings!!

I would like to ask a question about dedup
eg: |dedup host ,IP
|dedup host |dedup IP
I've tried but when I use a comma, dedup works only on the first fields, and I want that this can be performed on both sides not only one side, I wanted that the output for fields 1 and fields 2 no redundancy values come again?

for example:
|dedup host, IP --->this brings me the below output: and I want that this one could not be repeated also like on host fields.

host IP

x 1.1.1.1
y 1.1.1.1
z 2.2.2.2

what the best way to remove redundancy for two fields????????
I need your help?

Thanks!

Re: how can I use dedup command using many fields??

to4kawa — Thu, 19 Dec 2019 19:15:44 GMT

| stats count by your_dedup

Hi, @pacifikn
this is the easy way.

Re: how can I use dedup command using many fields??

woodcock — Thu, 19 Dec 2019 20:57:27 GMT

Like this:

...| dedup host
| dedup IP

Re: how can I use dedup command using many fields??

pacifikn — Fri, 20 Dec 2019 04:42:54 GMT

Thank you to4kawa and Woodcock,

Thank you for your assistance.

Re: how can I use dedup command using many fields??

bjcross — Fri, 20 Dec 2019 05:09:31 GMT

Using dedup on multiple fields with the comma isn't only working on the first field. It is actually removing events where the host and IP BOTH match.

Re: how can I use dedup command using many fields??

to4kawa — Fri, 20 Dec 2019 14:42:31 GMT

| makeresults 
| eval _raw="C IP
x 1.1.1.1
x 2.2.2.2
y 1.1.1.1
z 2.2.2.2"
| makemv delim="
" _raw
| multikv
| table C IP
| rename C as host
| dedup host
| dedup IP

| makeresults 
| eval _raw="C IP
x 1.1.1.1
x 2.2.2.2
y 1.1.1.1
z 2.2.2.2"
| makemv delim="
" _raw
| multikv
| table C IP
| rename C as host
| stats count by host IP
| table host IP

| makeresults 
| eval _raw="C IP
x 1.1.1.1
x 2.2.2.2
y 1.1.1.1
z 2.2.2.2"
| makemv delim="
" _raw
| multikv
| table C IP
| rename C as host
| dedup host,IP

Which result is correct?