All Apps and Add-ons

How to dedup based on other fields conditions?

kashtech
Observer

I am new and learning splunk, I have a 2 events like below with same event type. 

name='name1', user='abc', type='type1', other-fields     : latest event

name='name1', user='abc1', type='type1', other-fields  : past event

name='name2', user='def', type='type2', other-fields     

 

I want to dedup based on user field, but the dedup value changes but all other fields remain same. In this case I want to match fields name & type between first 2 events and pick up the latest one. 

My final filtered events should be:

name='xyz', user='abc', type='new', other-fields

name='name2', user='def', type='type2', other-fields

 

Any suggestions?

 

 

Labels (1)
Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @kashtech,

did you tried with the last option in stats?

<your_search>
| stats 
   last(name) AS name 
   last(type) AS type 
   last(other-fields) AS other-fields 
   BY user

Ciao.

Giuseppe

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

This is not possible given the provided information - there is no provided logical relationship between your dummy data and the expected output.

Please provide more accurate representations of your events, the expected output and  the logical relationship between them.

0 Karma

kashtech
Observer

Oh! sorry for the typo, my expected output:

name='xyz', user='abc', type='type1', other-fields

name='name2', user='def', type='type2', other-fields

0 Karma

Taruchit
Contributor

You mentioned in your description that you want to use dedup on user field.

If you check the data given, the first and second row have different field values for user: - abc, abc1.

In your actual dataset, are two values different or are they same?

0 Karma

Taruchit
Contributor

Hi @kashtech,

Your dataset's field values and the expected output field values are not seeming to be in sync.

For example: - 

Dataset that you shared: -

name='name1', user='abc', type='type1', other-fields     : latest event

name='name1', user='abc1', type='type1', other-fields  : past event

name='name2', user='def', type='type2', other-fields     

Here, the two distinct values for field "type" are: - type1 and type2.

 

However, in your expected result, the value of field "type" is new.

name='xyz', user='abc', type='new', other-fields

name='name2', user='def', type='type2', other-fields

 

Thus, for clarity it would be helpful if you could share the dataset: -

1. in tabular format

2. in sync between input and expected output.

 

Thank you

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...