I suggest the modular input cache events internally rather than try to fetch them from an index. If a new event is in the cache then it's a duplicate and can be discarded; otherwise, index it and add it to the cache. Yes, the cache will limit your look-back for duplicates, but will perform vastly better than scanning an index for every incoming event. You'll still need to handle duplicates at search time, but there will be far fewer of them.
--- If this reply helps you, an upvote would be appreciated.