topic Re: count uri_host hits including subdomains? Squid logs. in Getting Data In

count uri_host hits including subdomains? Squid logs.

mrgibbon — Mon, 05 Nov 2012 21:32:17 GMT

Hey Guys, Im trying to come up with some searches for our HR department.
We sometimes have to present them with evidence of web browsing activity for a named user.
I would like to know if there is a way of combining all the hits from web pages that are in the same domain?

eg. news.com.au also has resources.news.com.au and resources2.news.com.au etc

Is there a way of getting a really acurate portrail of a users browsing habits from Squid logs?

Thanks.

Re: count uri_host hits including subdomains? Squid logs.

Ayn — Tue, 06 Nov 2012 10:06:02 GMT

If you have a subdomain and just want to get a number of the hits, that's easy. In the Splunk for Squid app you could either use the Requests search view and simply put a wildcard before the domain you're interested in looking at in the Host field, for instance "*.news.com.au".

If on the other hand you're looking for some kind of generic mechanism for identifying which domains are simply subdomains of the ones you're interested in, that's simply impossible, because there's no way to know where a "main domain" ends and where a subdomain stops. Especially since different TLDs utilise different standards, so for example for the .com domain it might be interesting to break out the two last parts - "google.com", "splunk.com" etc - whereas for .au and .uk domains among others that won't make sense because you'll get "com.au", "co.uk", "ac.uk" and so on.

Re: count uri_host hits including subdomains? Squid logs.

martin_mueller — Tue, 06 Nov 2012 10:15:43 GMT

You could build a field from the full domain using regular expressions along with a list of TLDs. In untested rex guise it might look something like this:

... | rex field=domain "(?<base_domain>[^.]+)\.(com|co\.uk|com\.au|and|so|on))$" | ...

Such a list of TLDs should be obtainable, finite, and relatively stable.

Re: count uri_host hits including subdomains? Squid logs.

mrgibbon — Tue, 06 Nov 2012 10:37:13 GMT

Yeah, I really wanted to be able to bundle up all the sub-domains with the TLD's,
like, 3 hits on news.com.au plus 3 hits on resources.news.com.au = 6 hits.
Ive done a lot of looking around today, and there doesn't seem to be a good way of doing it.
Bummer.

Re: count uri_host hits including subdomains? Squid logs.

mrgibbon — Tue, 06 Nov 2012 10:38:13 GMT

Re: count uri_host hits including subdomains? Squid logs.

sowings — Tue, 06 Nov 2012 12:29:45 GMT

Not strictly impossible, but hard. The domain registrars of various countries have different rules about how entity names are joined with TLD components to produce a final domain name. You can start here on Wikipedia, but it could well be a moving target.