Splunk Search

how can I use dedup command using many fields??

pacifikn
Communicator

Greetings!!

I would like to ask a question about dedup
eg: |dedup host ,IP
|dedup host |dedup IP
I've tried but when I use a comma, dedup works only on the first fields, and I want that this can be performed on both sides not only one side, I wanted that the output for fields 1 and fields 2 no redundancy values come again?

for example:
|dedup host, IP --->this brings me the below output: and I want that this one could not be repeated also like on host fields.

host IP

x 1.1.1.1
y 1.1.1.1
z 2.2.2.2

what the best way to remove redundancy for two fields????????
I need your help?

Thanks!

Tags (2)
0 Karma
1 Solution

woodcock
Esteemed Legend

Like this:

...| dedup host
| dedup IP

View solution in original post

0 Karma

bjcross
Explorer

Using dedup on multiple fields with the comma isn't only working on the first field. It is actually removing events where the host and IP BOTH match.

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="C IP
x 1.1.1.1
x 2.2.2.2
y 1.1.1.1
z 2.2.2.2"
| makemv delim="
" _raw
| multikv
| table C IP
| rename C as host
| dedup host
| dedup IP

OR

| makeresults 
| eval _raw="C IP
x 1.1.1.1
x 2.2.2.2
y 1.1.1.1
z 2.2.2.2"
| makemv delim="
" _raw
| multikv
| table C IP
| rename C as host
| stats count by host IP
| table host IP

OR

| makeresults 
| eval _raw="C IP
x 1.1.1.1
x 2.2.2.2
y 1.1.1.1
z 2.2.2.2"
| makemv delim="
" _raw
| multikv
| table C IP
| rename C as host
| dedup host,IP

Which result is correct?

0 Karma

pacifikn
Communicator

Thank you to4kawa and Woodcock,

Thank you for your assistance.

woodcock
Esteemed Legend

Like this:

...| dedup host
| dedup IP
0 Karma

to4kawa
Ultra Champion
| stats count by your_dedup

Hi, @pacifikn
this is the easy way.

0 Karma
Get Updates on the Splunk Community!

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...