Hi,
How can I dedup one input to a combined search?
e.g;
index=dataA OR index=dataB | dedup <some field only present in dataB>
dataB has duplicate records, and I want to exclude only those records in dataB, by a field present in only those records.
try with fields
command to remove this fields before use it, like this:
index=dataA OR index=dataB | ...|fields -source_name_fields, host, ip, ....
Assuming that the field is only present in dataB, you could do:
| dedup <field only present in dataB> keepempty=true
This will keep unique values of that field plus all events where the field isn't present. See the docs on dedup for more specific detail, and other options.
Thanks, I did try that but it didn't seem to do the job. If I search just that source and dedup, fine. But if I include multiple sources, duplicate records reappeared. I am also piping the results to transaction command, and that may have an impact too.