topic Re: How to dedup based on other fields conditions? in All Apps and Add-ons

How to dedup based on other fields conditions?

kashtech — Sat, 20 May 2023 00:35:15 GMT

I am new and learning splunk, I have a 2 events like below with same event type.

name='name1', user='abc', type='type1', other-fields : latest event

name='name1', user='abc1', type='type1', other-fields : past event

name='name2', user='def', type='type2', other-fields

I want to dedup based on user field, but the dedup value changes but all other fields remain same. In this case I want to match fields name & type between first 2 events and pick up the latest one.

My final filtered events should be:

name='xyz', user='abc', type='new', other-fields

name='name2', user='def', type='type2', other-fields

Any suggestions?

Re: How to dedup based on other fields conditions?

Taruchit — Sat, 20 May 2023 02:59:03 GMT

Hi @kashtech,

Your dataset's field values and the expected output field values are not seeming to be in sync.

For example: -

Dataset that you shared: -

name='name1', user='abc', type='type1', other-fields : latest event

name='name1', user='abc1', type='type1', other-fields : past event

name='name2', user='def', type='type2', other-fields

Here, the two distinct values for field "type" are: - type1 and type2.

However, in your expected result, the value of field "type" is new.

name='xyz', user='abc', type='new', other-fields

name='name2', user='def', type='type2', other-fields

Thus, for clarity it would be helpful if you could share the dataset: -

1. in tabular format

2. in sync between input and expected output.

Thank you

Re: How to dedup based on other fields conditions?

kashtech — Sat, 20 May 2023 03:01:16 GMT

Oh! sorry for the typo, my expected output:

name='xyz', user='abc', type='type1', other-fields

name='name2', user='def', type='type2', other-fields

Re: How to dedup based on other fields conditions?

Taruchit — Sat, 20 May 2023 03:04:15 GMT

You mentioned in your description that you want to use dedup on user field.

If you check the data given, the first and second row have different field values for user: - abc, abc1.

In your actual dataset, are two values different or are they same?

Re: How to dedup based on other fields conditions?

ITWhisperer — Sat, 20 May 2023 05:57:56 GMT

This is not possible given the provided information - there is no provided logical relationship between your dummy data and the expected output.

Please provide more accurate representations of your events, the expected output and the logical relationship between them.

Re: How to dedup based on other fields conditions?

gcusello — Sun, 21 May 2023 05:44:31 GMT

Hi @kashtech,

did you tried with the last option in stats?

<your_search> | stats last(name) AS name last(type) AS type last(other-fields) AS other-fields BY user

Ciao.

Giuseppe