topic Re: How to remove duplicates from results of two searches on two different fields that have some of the same values? in Splunk Search

How to remove duplicates from results of two searches on two different fields that have some of the same values?

BaptVe — Tue, 24 May 2016 10:52:06 GMT

Hello !

I launch a search with append to put the results of two searches together on different fields, but then I would like to remove the duplicates on these results:

First LOG :

24/05/2016 11:33:19,719 (...) service id : one

one is the value of the field Service

Second LOG:

24/05/2016 11:38:33,688 (...) service id : two

two is the value of the field state

The two logs are written differently and these two service id have two different field names in Splunk.
I've appended the two results:

index=XXXX com="*xxxx*" service=*  | append [ search
index=XXXX com="*xxxx*"  state=* ] 
| where state!= service |stats list(state)

And I tried with where to show the list, but without success!

Any help is welcome 😄

Thanks !

Re: How to remove duplicates from results of two searches on two different fields that have some of the same values?

lukasz92 — Tue, 24 May 2016 11:05:31 GMT

Make one field common. For example add to the second search: 'eval service=state'
Use 'dedup service, index, com' (and all other fields that are the same, if there is a duplicate)

Re: How to remove duplicates from results of two searches on two different fields that have some of the same values?

ishaanshekhar — Tue, 24 May 2016 11:32:43 GMT

You can modify your search like this...

index=XXXX com="xxxx" service= | rename service as state 
|append [ search index=XXXX com="xxxx" state= ] 
| dedup state |stats list(state)

Thanks
Ishaan