Splunk Search

Extracting domain name out of a url

imarks004
Path Finder

I am trying to field extraction working for just domains accessed on my Ironport WSAs but am having an issue extracting just the domain piece out of a url.
For example, if I do a search by top s_hostname I get the following: 0.4.channel.facebook.com 0.52.channel.facebook.com 0.57.channel.facebook.com 0.chstatic.cvcdn.com 0.gvt0.com 0.media.dorkly.cvcdn.com 0.media.todaysbigthing.cvcdn.com 0.r.msn.com 0.tqn.com 0.track.ning.com I am trying to get a regex working to strip everything out to the left of the domain name so I would only see facebook.com and not 0.4.channel.facebook.com. I am not having any luck coming up with a regex to handle this.

Tags (2)

dariusdamalakas
New Member

Takes everything up until 3rd slash

rex field=Uri "^(?[^/]/[^/]/[^/]*)"

0 Karma

fwijnholds_splu
Splunk Employee
Splunk Employee

There's an App for that! The URL toolbox is my absolute fav but maybe URL Parse already does the trick?

Your SPL would look like this:

`method=GET| ut_parse(referer)`  

Make sure you use the back tick so Splunk knows you are calling a macro.

0 Karma

GeekMikeGrace
Engager

I ended up going with

\/\/(?:[^@\/\n]+@)?(?:www\.)?(?<refdomain>[^:\/\n]+)

Used it context it looks like

method=GET| rex field=referer "\/\/(?:[^@\/\n]+@)?(?:www\.)?(?<refdomain>[^:\/\n]+)"| stats values(refdomain)

See the extraction in action https://regex101.com/r/iVrIlL/1

tpflicke
Path Finder

To deal with all the various examples in this thread and all other possible cases such as new domains like .london, I think it will need something more than a reasonably short regex line.

I would probably go down the route of calling a Python script to deal with the cases to my satisfaction and being able to lay out the logic in a maintainable way. Maybe there is a splunk app or add-on that provides such functionality, if not, it could make a nice exercise to create one.

A few test cases:

conductor.io.com => io.com
support.expedia.co.uk => expedia.co.uk
0.52.channel.facebook.com => facebook.com
0.52.channel.facebook.london => facebook.london

stanleyglover
New Member

Extraction can easily be done by some simple steps given at http://www.perlmonks.org/?node_id=670802. Various formulas are also available that can easily extract domain name from the URL using Regex who’s examples you can see at above site too. After reading if still some query remains unsolved feel free to ask..

0 Karma

jrodriguezap
Contributor

This can also be even more efficient (if either com.br, com.pe, com.jo):

(?<_hostname>(\d{1,3}.\d{1,3}?|[^\.\s]+?)\.([^\.\s]{1,3}|[^\.\s]{1,3}\.[^\.\s]{1,3}))$

southeringtonp
Motivator

Assuming you always want only two levels:

| rex field=s_hostname "\.(?<s_domainname>\S+\.\S+)$"

dariusdamalakas
New Member

I downvoted this post because does not work anymore. n

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Probably a more efficient regex is: (?<s_domainname>[^\.\s]+\.[^\.\s]+)$ instead.

mstephenson716
Explorer

This worked for me.

0 Karma

mIliofotou_splu
Splunk Employee
Splunk Employee

I don think this works any more ...

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...