How can I dedup one part of a combined search?

[rjlohan]

Hi,

How can I dedup one input to a combined search?

e.g;

index=dataA OR index=dataB | dedup <some field only present in dataB>

dataB has duplicate records, and I want to exclude only those records in dataB, by a field present in only those records.

[fdi01]

try with fields command to remove this fields before use it, like this:

index=dataA OR index=dataB | ...|fields -source_name_fields, host, ip, ....

[acharlieh]

Assuming that the field is only present in dataB, you could do:

| dedup <field only present in dataB> keepempty=true

This will keep unique values of that field plus all events where the field isn't present. See the docs on dedup for more specific detail, and other options.

[rjlohan]

Thanks, I did try that but it didn't seem to do the job. If I search just that source and dedup, fine. But if I include multiple sources, duplicate records reappeared. I am also piping the results to transaction command, and that may have an impact too.