Can you help me with a query that removes duplicate events?

[maheshsat]

I want to suppress duplicate events. I am trying the below command, but I'm unable to remove duplicate events.

index="id_sp" sourcetype="current" | dedup _raw | search  Period = "Jun-17"  Prepare="Actual" Fixed ="12345"  | rex "(.sap=\"(?[^\",]+))" | table Period  

Note i don't have any fields in which value get changed.

[bjoernjensen]

Hey,

your table (at the end of your spl) has just Period as column. Since you do a dedup on _raw and search for a certain (no wildcard) Prepare there almost must be only dublicate entries. Moreover your search is optimizable, and maybe you want to add more columns to the table:

index="id_sp" sourcetype="current" Period = "Jun-17" Prepare="Actual" Fixed ="12345"
| dedup _raw
| rex ".sap=(?<sap>\"(?[^\",]+))"
| table Period Prepare Fixed sap

Cheerz,
Björn