How to detect url/domain category change in proxy logs

[vicky2903]

Hi Everyone,

I want to create a splunk query which can detect url/domain category change in the proxy logs within last 7 days

Example

Initial domain/url category

Domain/url : abc.com
Category : New Domain
Date : 12 May 2020

Final domain/url category

Domain/url : abc.com
Category : Business
Date : 18 May 2020

Kindly help at the earliest

Thanks

[PavelP · Accepted solution]

Hello @vicky2903

from the naive perspective this search will find domains with more than 1 category:

sourcetype=proxylog earliest=-7@d latest=now
| stats  dc(url_category) AS num_of_cat by domain 
|  where num_of_cat > 1

but in real life things are different:

• url categorized differently depending of the uri path, for example www.example.com/sport and www.example.com/chat can belong to different categories
• proxy can categorize url differently if DPI is applied (CONNECT www.example.com:443 vs https://www.example.com)
• url can be uncategorized if URL filter is bypassed
• url can belong to several categories (ex.: "Media Sharing, Streaming Media")

I suggest to compare not domains alone but domain + first segment of url to handle such cases.

Which URL database do use use? Cases above can be applied to Talos (Cisco), Trustedsource (McAfee) and BlueCoat/Symantec/Broadcom among others

[vicky2903]

Hi ,

Thanks for the answer

Thanks

[PavelP]

@vicky2903 please accept my answer if it resolve your query