While using the table for bro conn data, I am getting duplicate data; however, if I use mvdedup, I get all the desired results except the id.orig_h and id.resp_h. If I use mvdedup for these two entries, I get blank values.
index=bro_conn
| eval id.orig_h=mvdedup(id.orig_h)
| eval id.resp_h=mvdedup(id.resp_h)
| eval dest_port=mvdedup(dest_port)
| eval resp_ip_bytes=mvdedup(resp_ip_bytes)
| eval orig_ip_bytes=mvdedup(orig_ip_bytes)
| eval conn_state=mvdedup(conn_state)
| eval proto=mvdedup(proto)
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state
Here are the results if I don't use mvdedup:
Is there a better way to dedup the results, and if this is the right way, how can I make this work for the src and dest IPs?
Thank you!
Hi @geekf,
If complete event is duplicated, you can try dedup on _raw field;
index=bro_conn
| dedup _raw
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state
Otherwise try below;
index=bro_conn
| eval id.orig_h=mvdedup('id.orig_h')
| eval id.resp_h=mvdedup('id.resp_h')
| eval dest_port=mvdedup(dest_port)
| eval resp_ip_bytes=mvdedup(resp_ip_bytes)
| eval orig_ip_bytes=mvdedup(orig_ip_bytes)
| eval conn_state=mvdedup(conn_state)
| eval proto=mvdedup(proto)
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state
Hi @geekf,
If complete event is duplicated, you can try dedup on _raw field;
index=bro_conn
| dedup _raw
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state
Otherwise try below;
index=bro_conn
| eval id.orig_h=mvdedup('id.orig_h')
| eval id.resp_h=mvdedup('id.resp_h')
| eval dest_port=mvdedup(dest_port)
| eval resp_ip_bytes=mvdedup(resp_ip_bytes)
| eval orig_ip_bytes=mvdedup(orig_ip_bytes)
| eval conn_state=mvdedup(conn_state)
| eval proto=mvdedup(proto)
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state
Thank you.
dedup _raw didn't work, single quotes for src and dest worked fine.