How to use rex to extract values from URLs into a field?

[willamwar]

Hello all,

From the following list

http://www.foo.com:80/main.html
http://www.foo.com:80/xe/journal/v1/book/nF1.jpg
http://www.goo.com:80/fiction/journal/07/40

Where the website is foo, find journal and extract /xe/journal/
So there are other websites, and not every target website has journal, and xe can be an number of characters

I have a working regex

 https?:\/\/.+?(?=foo\.com)[^\/]++(\/?[^\/]+\/(?=journal)journal\/?)

My issue is I can't figure out how to get rex to send the data to the 'data' variable.

| rex field=trimurl "https?:\/\/.+?(?=foo\.com)[^\/]++(?<data>.*)(\/?[^\/]+\/(?=journal)journal\/?)" 

[woodcock]

Also, check out URL Toolbox:
https://splunkbase.splunk.com/app/2734/

[gokadroid · Accepted solution]

To get you the data in field data, rex part can be handled as follows:

rex field=trimurl "https?:\/\/.+?(foo\.com)[^\/]+(?<data>(\/[^\/]+){2}\/)"

See here the regex a work

If in field called data you specifically want the keyword journal together with variable number string called xe, where xe is one or more charaters long, like in the form /xe/journal/ then try this:

rex field=trimurl "https?:\/\/.+?(foo\.com)[^\/]+(?<data>(\/[^\/]+)\/journal\/)"

[gokadroid]

oh damn...thanks...if you can accept the answer for it to be closed then that will help too...editing the answer as per your need and to correct my mistake..

[willamwar]

Thank you so much.

Could you fix the spelling of feld --> field so that others don't get an error and have to figure that out.