Splunk Search

Problem using set diff together with mvexpand

kenliu
Explorer

I'm trying to compare multi-value fields from multiple events and display the diff between the two sets.

For example, I want to know if a new cookie was introduced today that wasn't there yesterday. Each access log event would contain multiple cookie key/value pairs. So I first use regex to extract all cookie names into an multi-valued field. Then use the mvexpand command to break up the mvfield into individual events, and finally dedup that field to get the list of distinct cookies. I put this search into two subsearches with different start/end times and use the set diff command to compare the two. The result isn't what I'm expecting. It's listing out all the cookies even when the cookie names are present in both sets. My search example:

| set diff 
[search index=access_log earliest=-1m@m latest=@m 
   | rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]*)" 
   | fields - _raw | fields cookieNames 
   | mvexpand cookieNames 
   | dedup cookieNames 
   | table cookieNames]
[search index=access_log earliest=-2m@m latest=-1m@m 
   | rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]*)" 
   | fields - _raw | fields cookieNames 
   | mvexpand cookieNames 
   | dedup cookieNames 
   | table cookieNames]

I'm suspecting this is not working because my list of cookieNames was created post-process using mvexpand and perhaps set diff only works on two list of raw events. Anyone has any experience solving this issue?

0 Karma
1 Solution

somesoni2
Revered Legend

Try like this (will give list of cookieNames field which were reported today but not yesterday.

index=access_log earliest=-1d@d latest=now
| eval Date=if(_time>=relative_time(now(),"@d"),"Today","Yesterday")
| rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]</em>)"
| stats Values(Date) as ReportedOn by cookieNames 
| where mvcount(ReportedOn)=1 AND ReportedOn="Today"

View solution in original post

somesoni2
Revered Legend

Try like this (will give list of cookieNames field which were reported today but not yesterday.

index=access_log earliest=-1d@d latest=now
| eval Date=if(_time>=relative_time(now(),"@d"),"Today","Yesterday")
| rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]</em>)"
| stats Values(Date) as ReportedOn by cookieNames 
| where mvcount(ReportedOn)=1 AND ReportedOn="Today"

kenliu
Explorer

That worked for me. Thanks!

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...