topic Re: How to use the dedup command for this search? in Splunk Search

How to use the dedup command for this search?

evinasco — Tue, 10 Apr 2018 14:57:30 GMT

Hi Team,

I have the next source list indexed in Splunk

I need to let in only the last source by each factory owner

**source**                                                                                                    -     **_time**

/var/log/gestion/logs/equipments/alcatel_ports.1519230396.log   -      2018-03-22 17:41:44
/var/log/gestion/logs/equipments/cisco_ports.1519230441.log          -      2018-03-22 17:41:44
/var/log/gestion/logs/equipments/huawei_ports.1519230476.log     -     2018-03-22 17:41:44
/var/log/gestion/logs/equipments/juniper_ports.1519230550.log    -     2018-03-22 17:41:44

I have tried to use dedup command but for this data doesn't work ok. Does someone know how can I do that?

Regards

Re: How to use the dedup command for this search?

adonio — Tue, 10 Apr 2018 16:52:18 GMT

hello there,

dedup does not work as it looks at the exact value for the field, and your values for each factory owner are different in the source
try and extract the factory owner from the source field and then stats on the latest
something like this:

... your search ... | rex field=source "\/equipments\/(?<factory_owner>\w+)"
| stats latest(factory_owner) as last_source_from_factory_owner by source

see also here:
https://regex101.com/r/uWt0Mf/2
hope it helps

Re: How to use the dedup command for this search?

DalJeanis — Tue, 10 Apr 2018 17:10:46 GMT

what do you mean by "let in"? Dedup has nothing whatsoever to do with ingestion of data.

Re: How to use the dedup command for this search?

evinasco — Tue, 10 Apr 2018 22:21:10 GMT

Ready

Thanks a lot