URL Toolbox problem with ut_parse_extended / ut_parse_extended_lookup

[ssteinmann]

Hi all,

i'm using URL Toolbox for analysing url entropy, shannon and meaning scoring is working also simple parse of url. I now want to extract the main url to match against some TI List but the ut_parse_extended(url, list) macro does not work. I'm using latest URl Toolbox Version 1.6 and Splunk 7.2.x.

This is my example Query:

| makeresults 
| eval url="www.domain.com"
| lookup ut_parse_simple_lookup url as url
| lookup ut_parse_extended_lookup url as url list as mozilla
| `ut_shannon(url)` 
| `ut_meaning(url)`

This does not include extractions from extended parse...any idea?

_time   url ut_fragment ut_meaning_ratio    ut_netloc   ut_params   ut_path ut_query    ut_scheme   ut_shannon
17.04.2019 11:49    www.domain.com  None    0.35714285714285715 www.domain.com  None    None    None    None    3.6978458230844127

all phyton scripts are there:

ll /opt/splunk/etc/apps/utbox/bin/
-rw-r--r-- 1 splunk splunk 359422 17. Apr 09:56 bayesian_bad.dic
-rw-r--r-- 1 splunk splunk 269354 17. Apr 09:56 bayesian_good.dic
-rw-r--r-- 1 splunk splunk  37289 17. Apr 09:56 meaning.dic
-rw-r--r-- 1 splunk splunk   7615 17. Apr 09:56 suffix_list_custom.dat
-rw-r--r-- 1 splunk splunk   7432 17. Apr 09:56 suffix_list_iana.dat
-rw-r--r-- 1 splunk splunk 157772 17. Apr 09:56 suffix_list_mozilla.dat
-rw-r--r-- 1 splunk splunk   2055 17. Apr 09:56 ut_bayesian.py
-rw-r--r-- 1 splunk splunk   2217 17. Apr 09:56 ut_countset.py
-rw-r--r-- 1 splunk splunk   1151 17. Apr 09:56 ut_levenshtein.py
-rw-r--r-- 1 splunk splunk    406 17. Apr 09:56 ut_log.py
-rw------- 1 splunk splunk    746 17. Apr 11:11 ut_log.pyc
-rw-r--r-- 1 splunk splunk   1802 17. Apr 09:56 ut_meaning.py
-rw-r--r-- 1 splunk splunk   1071 17. Apr 09:56 ut_parse_extended.py
-rw-r--r-- 1 splunk splunk   6199 17. Apr 09:56 ut_parse_lib.py
-rw------- 1 splunk splunk   6099 17. Apr 11:11 ut_parse_lib.pyc
-rw-r--r-- 1 splunk splunk    758 17. Apr 09:56 ut_parse_simple.py
-rw-r--r-- 1 splunk splunk   2405 17. Apr 09:56 ut_presets.py
-rw-r--r-- 1 splunk splunk    874 17. Apr 09:56 ut_shannon.py
-rw-r--r-- 1 splunk splunk    677 17. Apr 09:56 ut_suites.py

[ssteinmann]

Using the correct syntax fixed my problem...

| makeresults 
| eval url="http://sub.domain.com/folder/?test=ok", list="mozilla"
| `ut_parse_extended(url,list)`

Now all Fields are extracted from url.

_time,list,url,"ut_domain","ut_domain_without_tld","ut_fragment","ut_netloc","ut_params","ut_path","ut_port","ut_query","ut_scheme","ut_subdomain","ut_subdomain_count","ut_subdomain_level_1","ut_tld"
2019-04-23T11:45:40.000+0200,mozilla,"http://sub.domain.com/folder/?test=ok","domain.com",domain,None,"sub.domain.com",None,"/folder/",80,"test=ok",http,sub,1,sub,com