Eval recently collected events between indexes

strangelaw — Tue, 25 Oct 2016 11:13:58 GMT

Here is the thing:

I have 2 indexes: index_original and index_collected.

The plan is to compare/evaluate index_original events within selected timespan (say last 2 minutes) with index_collected events to see if there's any NEW EVENTS (e.g. not aleady collected to index_collected, that is) on index_original. I do not want to make one2one copy, but I want to compare with event (_time) or other unique identifier on that index if there's event available to be collected.

The target is that I have only fresh, newly collected events on index_collected - not whole stack again and then doing filtering/dedupping. Goal is to reduce index data amounts per each collection.

I have some other searches/table lookups to be done within same query, but this is the primary selector.

Any idea? Would be great help!

Re: Eval recently collected events between indexes

sundareshr — Tue, 25 Oct 2016 13:33:06 GMT

Will this work?

index=original OR index=collected | timechart list(_raw) as _raw by index | mvexpand _raw | where isnull(original) OR isnull(collected)

topic Eval recently collected events between indexes in Getting Data In

Eval recently collected events between indexes

Re: Eval recently collected events between indexes