How to remove duplicate events if all data is identical except the time and duration fields?

[asarran]

Hey, Fellow Splunkers

I have multiple duplicated events, all data on the event is identical to the exception of the time. I'm attempting to filter based on Alert ID; however, both events have the same alert id, but different times and Duration: for example:

Oct 31 00:16:50 alert: 123 Duration 200
Oct 31 00:18:50 alert: 123 Duration 300

Does Splunk have a compare operator to SORT the differences between the time or Duration which will help me eliminate the duplicates? The only concept the Duration could be random.

Thank You,

[ryhluc01]

They are not actually duplicates so you're going to want to do some research what your data is doing and why.

|dedup 1 day sortby day

You need to make sure that you specify which field you want your dedup to sort your data by.

[gokadroid]

An event "AlertId1,Duration1" if occurred at time T1 and at time T2, then for Splunk it is two different events as it happened at two different times. Keeping that in context a duplicate event will be when "AlertId1, Duration1, T1" occurs twice. This above scenario can be deduplicated by using dedup command as follows:

your base query to return fields alertId, duration
| dedup _time, alertId, duration
| complete your query

or if you want to deduplicate the data you are getting irrespective of the time it gets indexed at try

your base query to return events
| dedup _raw
| complete your query