Splunk Search

Problem using set diff together with mvexpand

kenliu
Explorer

I'm trying to compare multi-value fields from multiple events and display the diff between the two sets.

For example, I want to know if a new cookie was introduced today that wasn't there yesterday. Each access log event would contain multiple cookie key/value pairs. So I first use regex to extract all cookie names into an multi-valued field. Then use the mvexpand command to break up the mvfield into individual events, and finally dedup that field to get the list of distinct cookies. I put this search into two subsearches with different start/end times and use the set diff command to compare the two. The result isn't what I'm expecting. It's listing out all the cookies even when the cookie names are present in both sets. My search example:

| set diff 
[search index=access_log earliest=-1m@m latest=@m 
   | rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]*)" 
   | fields - _raw | fields cookieNames 
   | mvexpand cookieNames 
   | dedup cookieNames 
   | table cookieNames]
[search index=access_log earliest=-2m@m latest=-1m@m 
   | rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]*)" 
   | fields - _raw | fields cookieNames 
   | mvexpand cookieNames 
   | dedup cookieNames 
   | table cookieNames]

I'm suspecting this is not working because my list of cookieNames was created post-process using mvexpand and perhaps set diff only works on two list of raw events. Anyone has any experience solving this issue?

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Try like this (will give list of cookieNames field which were reported today but not yesterday.

index=access_log earliest=-1d@d latest=now
| eval Date=if(_time>=relative_time(now(),"@d"),"Today","Yesterday")
| rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]</em>)"
| stats Values(Date) as ReportedOn by cookieNames 
| where mvcount(ReportedOn)=1 AND ReportedOn="Today"

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Try like this (will give list of cookieNames field which were reported today but not yesterday.

index=access_log earliest=-1d@d latest=now
| eval Date=if(_time>=relative_time(now(),"@d"),"Today","Yesterday")
| rex max_match=100 field=Cookie "(^\b|;\s)(?<cookieNames>[^=]</em>)"
| stats Values(Date) as ReportedOn by cookieNames 
| where mvcount(ReportedOn)=1 AND ReportedOn="Today"

kenliu
Explorer

That worked for me. Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...