Problem using set diff together with mvexpand

[kenliu]

I'm trying to compare multi-value fields from multiple events and display the diff between the two sets.

For example, I want to know if a new cookie was introduced today that wasn't there yesterday. Each access log event would contain multiple cookie key/value pairs. So I first use regex to extract all cookie names into an multi-valued field. Then use the mvexpand command to break up the mvfield into individual events, and finally dedup that field to get the list of distinct cookies. I put this search into two subsearches with different start/end times and use the set diff command to compare the two. The result isn't what I'm expecting. It's listing out all the cookies even when the cookie names are present in both sets. My search example:

| set diff 
[search index=access_log earliest=-1m@m latest=@m 
   | rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]*)" 
   | fields - _raw | fields cookieNames 
   | mvexpand cookieNames 
   | dedup cookieNames 
   | table cookieNames]
[search index=access_log earliest=-2m@m latest=-1m@m 
   | rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]*)" 
   | fields - _raw | fields cookieNames 
   | mvexpand cookieNames 
   | dedup cookieNames 
   | table cookieNames]

I'm suspecting this is not working because my list of cookieNames was created post-process using mvexpand and perhaps set diff only works on two list of raw events. Anyone has any experience solving this issue?

[somesoni2 · Accepted solution]

Try like this (will give list of cookieNames field which were reported today but not yesterday.

index=access_log earliest=-1d@d latest=now
| eval Date=if(_time>=relative_time(now(),"@d"),"Today","Yesterday")
| rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]</em>)"
| stats Values(Date) as ReportedOn by cookieNames 
| where mvcount(ReportedOn)=1 AND ReportedOn="Today"

[kenliu]

That worked for me. Thanks!