Splunk Search

Matching a lookup table using rex or regex to match to data in the result of a query field

CyberCyberSec
Loves-to-Learn

Hello,

 

I am trying to use a lookup table to search against the URL field inside of the proxy logs. The use case is to find out if any users have been accessing any domains/URL's that is listed in the lookup file. I am trying to strip away all the extra characters and keep the url/domain that it is matching on.

This is what I have so far:

 

index=proxy sourcetype=proxy:syslog:proxy_web_policy[|inputlookup lookupfile_last_status  |return 1000 $lookup_domain_name]|rex field=proxy_url_field "(?<lookup_domain_name>(\w+\.)+\w+)"|table name url status

Example:

www.google.com/complete/search?client=chrome-omni&gs_ri=chrome-ext-ansg&xssi=t&q=gist.github.com/mcaj-admin/18558a1ec6a782d2452f971e806230c6&oit=3&url=https://gist.github.com/mcaj-admin/18558a1ec6a782d2452f971e806230c6&pgcl=4&gs_rn=42&psi=VqC6Zso_KQ3pgOGL&sugkey=AIzaSyBOti4mM-6x9WDnZIjIeyEU21OpBXqWBgw  

is is matching on "gist.github.com/mcaj-admin/18558a1ec6a782d2452f971e806230c6"

How can I strip away everything but what it is matching on? I am trying to figure out how to use regex + the lookup

 

Thanks.

 

Labels (1)
0 Karma

thambisetty
SplunkTrust
SplunkTrust

remove [#This will extract domain from url] from below search.

index=proxy sourcetype=proxy:syslog:proxy_web_policy 
| stats count as event_count by url
| rex field=url "^(\w+:\/\/)?(?<domain>[^\/]+)" #This will extract domain from url
| append [|inputlookup lookupfile_last_status | stats count as lookup_count by lookup_domain_name  | rename lookup_domain_name as domain  ]
| stats values(*) as * by domain
| where isnotnull(event_count) AND isnotnull(lookup_count)

 

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

Splunk Decoded: Business Transactions vs Business IQ

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, ...

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...