Splunk Search

How to generate a regular expression on a URL to capture a resource path and end on a optional parameter?

bcatwork
Path Finder

Hi all, I am looking for some help for the following use case.

I have a series of endpoints represented by full URLs logged across a few sources, of which i am trying to normalize to then aggregate on.

I am looking for the resource path, less any optional params. To say, I want to capture everything after the [//] double slash, domain name, first [/] singular slash and end that capture on an optional param [?]

https://answers.splunk.com/answers/ask.html?foo=bar --> Becomes --> answers/ask.html
https://answers.splunk.com/answers/ask.html --> Becomes --> answers/ask.html
http://docs.splunk.com/Documentation --> Becomes --> Documentation

0 Karma

gokadroid
Motivator

If say your url is already in the field myUrl, then try this:

yourBasequery to get myUrl field
|rex field=myUrl "http(s)*\:\/\/([^\/]+)\/(?<uri>[^\?\s]+)"

OR, try on _raw

yourBasequery to get url field
|rex field=_raw "http(s)*\:\/\/([^\/]+)\/(?<uri>[^\?\s]+)"
0 Karma

pgreer_splunk
Splunk Employee
Splunk Employee
rex field=urlField "^[^\/]+\/\/[^\/]+\/(?P<wantedField>[^\s;]+).*"

should pick up all three of your example use cases into the new extracted field named 'wantedField'

0 Karma

somesoni2
Revered Legend

You can try the replace OR rex-sed method to update the url field per your guideline. (sample run anywhere sample)

| gentimes start=-1 | eval url="https://answers.splunk.com/answers/ask.html?foo=bar https://answers.splunk.com/answers/ask.html http://docs.splunk.com/Documentation"; | makemv url | table url | mvexpand url 
| eval url=replace(url,"^[^\/]+\/\/[^\/]+\/([^\s\?;=]+).*","\1") | ...rest of the query

OR

| gentimes start=-1 | eval url="https://answers.splunk.com/answers/ask.html?foo=bar https://answers.splunk.com/answers/ask.html http://docs.splunk.com/Documentation"; | makemv url | table url | mvexpand url 
| rex mode=sed field=url "s/^[^\/]+\/\/[^\/]+\/([^\s\?;=]+).*/\1/" | ...rest of the query

aaraneta_splunk
Splunk Employee
Splunk Employee

Hi @bcatwork - I saw that you up-voted this answer from somesoni2. If this answer did help to solve your question, please don't forget to click "Accept" below the answer to close out this post. If not, please leave a comment with more feedback. Thanks!

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...