Event correlation: how to find only events _not_ referenced by others?

[marios_kstone]

Consider the following three events:

1: time=xxxx,sent=Item1,recd="Item0"
2: time=xxxx,sent=Item2,recd="Item1,Item3,Item7"
3: time=xxxx,sent=Item8,recd="Item9,Item8"

I want to be able to extract which Items have been sent but never received.
In the above example, would be Item2 (it never appears in "recd" field).

• Each record can hold a single sent item, but multiple receive items (it is already parsed as a mv field).
• Log for a received item can be in an event before the "sent" item log.

I was considering using transactions, but I am not sure about it. I am banging my head on this issue and would really appreciate any idea.

Thank you

[Ayn · Accepted solution]

Ok, so you have two sets, and you want the differences between them. You could either use set diff or a subsearch. Let's say your base search is "sourcetype=interestingevents".

set diff:

| set diff [search sourcetype=interestingevents | fields sent | fields - _*] [search sourcetype=interestingevents | fields recd | fields - _*]

This will give you the difference. If you know that all differing events will be the ones that have been sent but never received (and never the other way around), this should be enough.

If for some odd reason you do have events that were received but never sent, you could determine which "direction" the missing events are in by searching either for events that were sent but not received, or events that were received but not sent.

Sent, not received:

sourcetype=interestingevents NOT [search sourcetype=interestingevents | rename recd as sent | fields sent]

Received, not sent:

sourcetype=interestingevents NOT [search sourcetype=interestingevents | rename sent as recd | fields recd]

[marios_kstone]

Works like a charm. Thank you.

[marios_kstone]

Well, let's say that checking in a month timeframe should be acceptable. No other constraints.
Right now we are able to do this on a relational DB (not that easy there too) but since the number of records is high and the query is complex, it's taking just too much time to return results.

[Ayn]

How do you define the scope? Is there a time constraint in which you want to check for items sent but not received?