topic Re: How can I dedup one part of a combined search? in Splunk Search

How can I dedup one part of a combined search?

rjlohan — Wed, 24 Jun 2015 01:26:01 GMT

Hi,

How can I dedup one input to a combined search?

e.g;

index=dataA OR index=dataB | dedup <some field only present in dataB>

dataB has duplicate records, and I want to exclude only those records in dataB, by a field present in only those records.

Re: How can I dedup one part of a combined search?

acharlieh — Wed, 24 Jun 2015 03:36:15 GMT

Assuming that the field is only present in dataB, you could do:

| dedup <field only present in dataB> keepempty=true

This will keep unique values of that field plus all events where the field isn't present. See the docs on dedup for more specific detail, and other options.

Re: How can I dedup one part of a combined search?

rjlohan — Wed, 24 Jun 2015 05:46:42 GMT

Thanks, I did try that but it didn't seem to do the job. If I search just that source and dedup, fine. But if I include multiple sources, duplicate records reappeared. I am also piping the results to transaction command, and that may have an impact too.

Re: How can I dedup one part of a combined search?

fdi01 — Wed, 24 Jun 2015 09:33:28 GMT

try with fields command to remove this fields before use it, like this:

index=dataA OR index=dataB | ...|fields -source_name_fields, host, ip, ....