Splunk Search

Duplicate Entries in Table

geekf
Path Finder

While using the table for bro conn data, I am getting duplicate data; however, if I use mvdedup, I get all the desired results except the id.orig_h and id.resp_h. If I use mvdedup for these two entries, I get blank values.

 

 index=bro_conn
| eval id.orig_h=mvdedup(id.orig_h) 
| eval id.resp_h=mvdedup(id.resp_h)
| eval dest_port=mvdedup(dest_port) 
| eval resp_ip_bytes=mvdedup(resp_ip_bytes) 
| eval orig_ip_bytes=mvdedup(orig_ip_bytes) 
| eval conn_state=mvdedup(conn_state) 
| eval proto=mvdedup(proto) 
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

bro_conn-blnk.png

 

 

 

Here are the results if I don't use mvdedup:

bro_conn-dup.png

 

 

 

Is there a better way to dedup the results, and if this is the right way, how can I make this work for the src and dest IPs?

Thank you!

Labels (4)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @geekf,

If complete event is duplicated, you can try dedup on _raw field;

 index=bro_conn
| dedup _raw
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

Otherwise try below;

 index=bro_conn
| eval id.orig_h=mvdedup('id.orig_h') 
| eval id.resp_h=mvdedup('id.resp_h')
| eval dest_port=mvdedup(dest_port) 
| eval resp_ip_bytes=mvdedup(resp_ip_bytes) 
| eval orig_ip_bytes=mvdedup(orig_ip_bytes) 
| eval conn_state=mvdedup(conn_state) 
| eval proto=mvdedup(proto) 
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @geekf,

If complete event is duplicated, you can try dedup on _raw field;

 index=bro_conn
| dedup _raw
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

Otherwise try below;

 index=bro_conn
| eval id.orig_h=mvdedup('id.orig_h') 
| eval id.resp_h=mvdedup('id.resp_h')
| eval dest_port=mvdedup(dest_port) 
| eval resp_ip_bytes=mvdedup(resp_ip_bytes) 
| eval orig_ip_bytes=mvdedup(orig_ip_bytes) 
| eval conn_state=mvdedup(conn_state) 
| eval proto=mvdedup(proto) 
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

geekf
Path Finder

Thank you. 

dedup _raw didn't work, single quotes for src and dest worked fine.

0 Karma
Get Updates on the Splunk Community!

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...