Dedup not removing results that are duplicates

[Southy567]

Hi All!

Had a look around but couldn't find an answer to this. I'm trying to do a search where I track a users log in journey leading to a specific failed attempt error. The logging system doubles up on events so i'm only looking for values that happen at different times, and remove the duplicates that show as occurring at the exact same time.

However, my search keeps showing all the events and ignoring the dedup in my search and I cannot for the life of me figure out why. Example of search below:

index=INDEX sourcetype=SOURCETYPE <Search Phrase>
| eval LockoutTime=strftime(_time,"%Y-%m-%d %H:%M:%S %Z")
| transaction USERID maxspan=30M mvlist=true endswith=(EventDescription=EVENT)
| table LockoutTime USERID EventDescription Message EventCode Result
| dedup 1 LockoutTime
| where mvcount(EventCode)>1

Any help would be greatly appreciated.

[yuanliu]

Several pointers.  First, if every event is logged twice, why don't you dedup the events?  Instead, the code you showed dedup after expensive transaction command.  Second, why dedup a text field when the text field is made from _time?  If anything, the following will be more efficient.

index=INDEX sourcetype=SOURCETYPE <Search Phrase>
| dedup _time ``` default count is 1 ```
| transaction USERID maxspan=30M mvlist=true endswith=(EventDescription=EVENT)
| where mvcount(EventCode)>1
| eval LockoutTime=strftime(_time,"%Y-%m-%d %H:%M:%S %Z")
| table LockoutTime USERID EventDescription Message EventCode Result

If this still gives you seemingly duplicate outputs, you need to look at those transactions carefully to find out subtle differences