topic Re: How to trim base domain from url in Getting Data In

How to trim base domain from url

jgrantham — Mon, 08 Jul 2019 20:41:11 GMT

I have several pieces of data that look like this:

subdomain1.domain.com
subdomain2.domain.com

Question is how do I only pull the domain.com part in a Splunk search?

Re: How to trim base domain from url

richgalloway — Mon, 08 Jul 2019 22:52:11 GMT

There are a few ways to do that. The URL Toolbox app adds custom commands that will parse URLs. There's also the URL Parser app, but I have no experience with it.

If your data is not too complex, you can also parse it yourself using the rex command. This puts the domain.com part into the 'domain' field, but may need to be adjusted to suit your real data.

... | rex ".*?\.(?<domain>.*)" | ...

Re: How to trim base domain from url

jgrantham — Tue, 09 Jul 2019 12:01:46 GMT

Here is what I am using:

| rex field=dest_host".?.(?.)"
|table domain

What am I doing wrong? dest_host contains data like www. domain.com or sub.domain.com. I want to pull a count of domain.com. I am new to splunk and can't figure this out.

Any help would be greatly appreciated.

Re: How to trim base domain from url

richgalloway — Tue, 09 Jul 2019 12:16:01 GMT

Please edit your comment to indent the SPL at least 4 spaces so the SPL is preserved. Tell us what you get from the SPL and what you expected to see.

Re: How to trim base domain from url

jgrantham — Wed, 30 Sep 2020 01:15:59 GMT

Ok. I am running a query where one of the fields is dest_host. This will bring back results like www.domain.com, sub.domain.com, sub1.domain.com. I am trying to get a count based on the total using everything that includes domain.com. I currently have to do this manually and it is a pain.

Here is the SPL:
index=us_cseo_prod_webproxy sourcetype=mcafee:wg:kv action_name=allow policydecidingaccess="Allow Hosts in Global Whitelist - Telephone Directories"
| table user dest
| stats count by dest

What I get is a table with the following:
dest count
1 static.yellowpages.ca 19
2 www.yellowpages.ca 2

What I would like to see is :
yellowpages.ca 21

Re: How to trim base domain from url

jgrantham — Tue, 09 Jul 2019 12:59:21 GMT

Here is what I am using:

    index=us_cseo_prod_webproxy sourcetype=mcafee:wg:kv action_name=allow policydecidingaccess="Allow Hosts in Global Whitelist - Telephone Directories" |  table user dest|  stats count by dest

Here is what I get back:

dest    count

1 static.yellowpages.ca 19
2 www.yellowpages.ca 2

Here is what I would like to see:

dest count
yellowpages.ca 21

Re: How to trim base domain from url

jgrantham — Tue, 09 Jul 2019 13:59:45 GMT

Finally got it to work. Here is what I came up with:

     index=us_cseo_prod_webproxy sourcetype=mcafee:wg:kv action_name=allow policydecidingaccess="Allow Hosts in Global Whitelist - Telephone Directories" | rex field=dest_host "\.(?<domainname>\S+.\S+)$" |table domainname | stats count by domainname

Here is output:

    domainname  count
   whitepages.com   2
    yellowpages.ca  21