Eval recently collected events between indexes

[strangelaw]

Here is the thing:

I have 2 indexes: index_original and index_collected.

The plan is to compare/evaluate index_original events within selected timespan (say last 2 minutes) with index_collected events to see if there's any NEW EVENTS (e.g. not aleady collected to index_collected, that is) on index_original. I do not want to make one2one copy, but I want to compare with event (_time) or other unique identifier on that index if there's event available to be collected.

The target is that I have only fresh, newly collected events on index_collected - not whole stack again and then doing filtering/dedupping. Goal is to reduce index data amounts per each collection.

I have some other searches/table lookups to be done within same query, but this is the primary selector.

Any idea? Would be great help!

[sundareshr · Accepted solution]

Will this work?

index=original OR index=collected | timechart list(_raw) as _raw by index | mvexpand _raw | where isnull(original) OR isnull(collected)