https://community.splunk.com/t5/Splunk-Search/Need-help-with-Dedup-while-extracting-fields/m-p/227232#M67114 <P>Try this</P> <PRE><CODE>base search | extract mv_add=t pairdelim="," kvdelim="=:" |eval partnerId=mvdedup(partnerId | eval ActivationId=mvdedup(ActivationId) | table ActivationId partnerId </CODE></PRE> Sat, 01 Oct 2016 19:05:23 GMT sundareshr 2016-10-01T19:05:23Z https://community.splunk.com/t5/Splunk-Search/Need-help-with-Dedup-while-extracting-fields/m-p/227231#M67113 <P>Base String is -----------------</P> <H2>OfferRedeemedRequest [partnerID=1234, partnerName=MCenter, messagePriority=9, userID=2a28bc-119d7597, channel=rest, offerIds=1bf6-16a0fdd59fc4, redemptionDate=2016-05-14T11:56:24+0000, productBarCode=null, productBarCodeType=null, productQuantity=null, productCost=null, productDiscount=null, transactionID=null, posID=null, storeID=null, storeName=null, correlationID=null, JmsHeaderPriority=4, JMSPriority=4, destination=PARTNER_REDEMPTION_CONNECTOR_SENDER_QUEUE, bannerOmsPid=151, bodyPayload={"offerRedemptionProductDetails":[{"productBarCode":"342423409351","productBarCodeType":"gtin","productQuantity":1,"productCost":3.99,"actualDiscount":"-5.0"}],"offerIds":"1bf8b493-e6a0fdd59fc4","timeZoneOffset":0,"appId":"30cbd0ce-9ae4-44dc-a783-1f391942874d","bannerOmsPid":18651,"eventType":"OfferRedeemed","partnerId":"14p59-11e2-8b8b-0800200c9p69","userId":"2a2b88e0-e8bc-0050569d7597","priority":9,"channel":"rest","details":{"offerIdType":"offerId","location":"","OfferGs1":"8110003025902520535001101503161231","Void":"No","ProductQty":"0","activationDate":"2016-03-13T01:37:42+0000","OfferState":"Redeemed","OfferCode":"193218370","deviceId":"","correlationId":"4090914257","HttpUserAgent":"Jakarta Commons-HttpClient/3.0.1","ProductPrice":"0.0","RedemptionDate":"2016-03-14T11:56:24+0000","TransactionId":"11297895292","ActivationId":"2db41330-e8bc-11e5-b2eb-005056b94bc4","channel":"ECOMM"},"timestamp":"Mar 14, 2016 4:58:24 AM","partnerName":"MCenter"}]</H2> <P>Use this command to extract fields for above string<BR /> | extract mv_add=t pairdelim="," kvdelim="=:" | table fields<BR /> . However in a single transaction, there is duplicates of the same field like as you see in the above string OfferIds gets repeated, partnerId gets repeated. I just need only once in the output .</P> <P>When I try to dedup offerId, partnerId after extracting and then table, it does not work. Any help would be appreciated to remove the duplicates<BR /> Tried<BR /> base search | extract mv_add=t pairdelim="," kvdelim="=:" |dedup ActivationId,partnerId,userId,offerIds,activationType,activationDate,channel | table ActivationId,partnerId,userId,offerIds,activationType,activationDate,channel </P> Tue, 29 Sep 2020 11:15:29 GMT https://community.splunk.com/t5/Splunk-Search/Need-help-with-Dedup-while-extracting-fields/m-p/227231#M67113 arunsubram 2020-09-29T11:15:29Z https://community.splunk.com/t5/Splunk-Search/Need-help-with-Dedup-while-extracting-fields/m-p/227232#M67114 <P>Try this</P> <PRE><CODE>base search | extract mv_add=t pairdelim="," kvdelim="=:" |eval partnerId=mvdedup(partnerId | eval ActivationId=mvdedup(ActivationId) | table ActivationId partnerId </CODE></PRE> Sat, 01 Oct 2016 19:05:23 GMT https://community.splunk.com/t5/Splunk-Search/Need-help-with-Dedup-while-extracting-fields/m-p/227232#M67114 sundareshr 2016-10-01T19:05:23Z