Duplicate Entries in Table

[geekf]

While using the table for bro conn data, I am getting duplicate data; however, if I use mvdedup, I get all the desired results except the id.orig_h and id.resp_h. If I use mvdedup for these two entries, I get blank values.

 index=bro_conn
| eval id.orig_h=mvdedup(id.orig_h) 
| eval id.resp_h=mvdedup(id.resp_h)
| eval dest_port=mvdedup(dest_port) 
| eval resp_ip_bytes=mvdedup(resp_ip_bytes) 
| eval orig_ip_bytes=mvdedup(orig_ip_bytes) 
| eval conn_state=mvdedup(conn_state) 
| eval proto=mvdedup(proto) 
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

bro_conn-blnk.png

Here are the results if I don't use mvdedup:

bro_conn-dup.png

Is there a better way to dedup the results, and if this is the right way, how can I make this work for the src and dest IPs?

Thank you!

[scelikok · Accepted solution]

Hi @geekf,

If complete event is duplicated, you can try dedup on _raw field;

 index=bro_conn
| dedup _raw
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

Otherwise try below;

 index=bro_conn
| eval id.orig_h=mvdedup('id.orig_h') 
| eval id.resp_h=mvdedup('id.resp_h')
| eval dest_port=mvdedup(dest_port) 
| eval resp_ip_bytes=mvdedup(resp_ip_bytes) 
| eval orig_ip_bytes=mvdedup(orig_ip_bytes) 
| eval conn_state=mvdedup(conn_state) 
| eval proto=mvdedup(proto) 
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

[geekf]

Thank you. 

dedup _raw didn't work, single quotes for src and dest worked fine.