topic Regex searching using a csv list vs inline RegEx in Splunk Search

Regex searching using a csv list vs inline RegEx

justinbarta — Wed, 31 May 2017 15:47:53 GMT

My search results return a list of FQDN domain names. I need to replace that domain name with an app name when a portion of that domain name is located.

For example this would be the output from my index
office365.com
outlook.office365.com
sharepoint.office365.com
office.com
fbcdn.net

This is my current search which works but becomes extremely slow when adding a lot of match lines. This search looks for any domain even if it contains subdomains.
For example X10232.fbserver.fbcdn.net. The Facebook line below would match on any *.fbcdn.net domain.
I also need to match on exacts like facebook.com

index=weblogs | sort 0 -domain
| eval domain=case(

match(domain,"^(?=.*\bwordpress.com\b).*$"),"WordPress",
match(domain,"^(?=.*\b.sharepoint.com\b).*$"),"Microsoft Office 365 - Sharepoint",
match(domain,"^(?=.*\b.office365.com\b).*$"),"Microsoft Office 365",
match(domain,"^(?=.*\b.fbcdn.net\b).*$"),"Facebook",
match(domain,"^(?=.*\b.facebook.com\b).*$"),"Facebook",
match(domain,"^(?=.*\bfacebook.com\b).*$"),"Facebook",

true(),domain) | table domain

I would like preform the same matching task but from a CSV lookup list with three fields. This way I can create two new fields for each event containing appname and apptype

Domain,AppName,AppType
facebook.com,Facebook,Social Media
office.com,Office 365,Productivity
fbcdn.net,Facebook,Social Media

Re: Regex searching using a csv list vs inline RegEx

micahkemp — Wed, 31 May 2017 16:17:06 GMT

You can do this with a lookup, making use of match_type (transforms.conf😞

match_type = <string>
* A comma and space-delimited list of <match_type>(<field_name>)
  specification to allow for non-exact matching
* The available match_type values are WILDCARD, CIDR, and EXACT.  EXACT is
  the default and does not need to be specified.  Only fields that should
  use WILDCARD or CIDR matching should be specified in this list

You probably want your lookup to look like:

     Domain,AppName,AppType
     facebook.com,Facebook,Social Media
     *.facebook.com,Facebook,Social Media
     office.com,Office 365,Productivity
     *.office.com,Office 365,Productivity
     fbcdn.net,Facebook,Social Media
     *.fbcdn.net,Facebook,Social Media

In order to match facebook.com and www.facebook.com, but not fakefacebook.com.

Re: Regex searching using a csv list vs inline RegEx

woodcock — Wed, 31 May 2017 17:01:46 GMT

It is not going to be as simple as you might expect; there are apps for this stuff and I would highly suggest you check out how others have tackled this same kind of thing (the lookup part is easy once you get the domain normalization stuff working):

URL Tollbox: https://splunkbase.splunk.com/app/2734/
URL Parser: https://splunkbase.splunk.com/app/1545/
URL Expander: https://splunkbase.splunk.com/app/3460/
URL Parser: https://splunkbase.splunk.com/app/3396/