Splunk Search

How can I dedup one part of a combined search?

rjlohan
Explorer

Hi,

How can I dedup one input to a combined search?

e.g;

index=dataA OR index=dataB | dedup <some field only present in dataB>

dataB has duplicate records, and I want to exclude only those records in dataB, by a field present in only those records.

Tags (1)
0 Karma

fdi01
Motivator

try with fields command to remove this fields before use it, like this:

index=dataA OR index=dataB | ...|fields -source_name_fields, host, ip, ....
0 Karma

acharlieh
Influencer

Assuming that the field is only present in dataB, you could do:

| dedup <field only present in dataB> keepempty=true

This will keep unique values of that field plus all events where the field isn't present. See the docs on dedup for more specific detail, and other options.

0 Karma

rjlohan
Explorer

Thanks, I did try that but it didn't seem to do the job. If I search just that source and dedup, fine. But if I include multiple sources, duplicate records reappeared. I am also piping the results to transaction command, and that may have an impact too.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...