All Apps and Add-ons

How to dedup based on other fields conditions?

kashtech
Observer

I am new and learning splunk, I have a 2 events like below with same event type. 

name='name1', user='abc', type='type1', other-fields     : latest event

name='name1', user='abc1', type='type1', other-fields  : past event

name='name2', user='def', type='type2', other-fields     

 

I want to dedup based on user field, but the dedup value changes but all other fields remain same. In this case I want to match fields name & type between first 2 events and pick up the latest one. 

My final filtered events should be:

name='xyz', user='abc', type='new', other-fields

name='name2', user='def', type='type2', other-fields

 

Any suggestions?

 

 

Labels (1)
Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @kashtech,

did you tried with the last option in stats?

<your_search>
| stats 
   last(name) AS name 
   last(type) AS type 
   last(other-fields) AS other-fields 
   BY user

Ciao.

Giuseppe

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

This is not possible given the provided information - there is no provided logical relationship between your dummy data and the expected output.

Please provide more accurate representations of your events, the expected output and  the logical relationship between them.

0 Karma

kashtech
Observer

Oh! sorry for the typo, my expected output:

name='xyz', user='abc', type='type1', other-fields

name='name2', user='def', type='type2', other-fields

0 Karma

Taruchit
Contributor

You mentioned in your description that you want to use dedup on user field.

If you check the data given, the first and second row have different field values for user: - abc, abc1.

In your actual dataset, are two values different or are they same?

0 Karma

Taruchit
Contributor

Hi @kashtech,

Your dataset's field values and the expected output field values are not seeming to be in sync.

For example: - 

Dataset that you shared: -

name='name1', user='abc', type='type1', other-fields     : latest event

name='name1', user='abc1', type='type1', other-fields  : past event

name='name2', user='def', type='type2', other-fields     

Here, the two distinct values for field "type" are: - type1 and type2.

 

However, in your expected result, the value of field "type" is new.

name='xyz', user='abc', type='new', other-fields

name='name2', user='def', type='type2', other-fields

 

Thus, for clarity it would be helpful if you could share the dataset: -

1. in tabular format

2. in sync between input and expected output.

 

Thank you

0 Karma
Get Updates on the Splunk Community!

What’s New & Next in Splunk SOAR

Security teams today are dealing with more alerts, more tools, and more pressure than ever.  Join us on ...

Your Voice Matters! Help Us Shape the New Splunk Lantern Experience

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

September Community Champions: A Shoutout to Our Contributors!

As we close the books on another fantastic month, we want to take a moment to celebrate the people who are the ...