Splunk Search

Duplicate Entries in Table

geekf
Path Finder

While using the table for bro conn data, I am getting duplicate data; however, if I use mvdedup, I get all the desired results except the id.orig_h and id.resp_h. If I use mvdedup for these two entries, I get blank values.

 

 index=bro_conn
| eval id.orig_h=mvdedup(id.orig_h) 
| eval id.resp_h=mvdedup(id.resp_h)
| eval dest_port=mvdedup(dest_port) 
| eval resp_ip_bytes=mvdedup(resp_ip_bytes) 
| eval orig_ip_bytes=mvdedup(orig_ip_bytes) 
| eval conn_state=mvdedup(conn_state) 
| eval proto=mvdedup(proto) 
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

bro_conn-blnk.png

 

 

 

Here are the results if I don't use mvdedup:

bro_conn-dup.png

 

 

 

Is there a better way to dedup the results, and if this is the right way, how can I make this work for the src and dest IPs?

Thank you!

Labels (4)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @geekf,

If complete event is duplicated, you can try dedup on _raw field;

 index=bro_conn
| dedup _raw
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

Otherwise try below;

 index=bro_conn
| eval id.orig_h=mvdedup('id.orig_h') 
| eval id.resp_h=mvdedup('id.resp_h')
| eval dest_port=mvdedup(dest_port) 
| eval resp_ip_bytes=mvdedup(resp_ip_bytes) 
| eval orig_ip_bytes=mvdedup(orig_ip_bytes) 
| eval conn_state=mvdedup(conn_state) 
| eval proto=mvdedup(proto) 
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @geekf,

If complete event is duplicated, you can try dedup on _raw field;

 index=bro_conn
| dedup _raw
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

Otherwise try below;

 index=bro_conn
| eval id.orig_h=mvdedup('id.orig_h') 
| eval id.resp_h=mvdedup('id.resp_h')
| eval dest_port=mvdedup(dest_port) 
| eval resp_ip_bytes=mvdedup(resp_ip_bytes) 
| eval orig_ip_bytes=mvdedup(orig_ip_bytes) 
| eval conn_state=mvdedup(conn_state) 
| eval proto=mvdedup(proto) 
| table _time id.orig_h id.resp_h dest_port orig_ip_bytes resp_ip_bytes conn_state

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

geekf
Path Finder

Thank you. 

dedup _raw didn't work, single quotes for src and dest worked fine.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...