Hi all,
i'm using URL Toolbox for analysing url entropy, shannon and meaning scoring is working also simple parse of url. I now want to extract the main url to match against some TI List but the ut_parse_extended(url, list)
macro does not work. I'm using latest URl Toolbox Version 1.6 and Splunk 7.2.x.
This is my example Query:
| makeresults
| eval url="www.domain.com"
| lookup ut_parse_simple_lookup url as url
| lookup ut_parse_extended_lookup url as url list as mozilla
| `ut_shannon(url)`
| `ut_meaning(url)`
This does not include extractions from extended parse...any idea?
_time url ut_fragment ut_meaning_ratio ut_netloc ut_params ut_path ut_query ut_scheme ut_shannon
17.04.2019 11:49 www.domain.com None 0.35714285714285715 www.domain.com None None None None 3.6978458230844127
all phyton scripts are there:
ll /opt/splunk/etc/apps/utbox/bin/
-rw-r--r-- 1 splunk splunk 359422 17. Apr 09:56 bayesian_bad.dic
-rw-r--r-- 1 splunk splunk 269354 17. Apr 09:56 bayesian_good.dic
-rw-r--r-- 1 splunk splunk 37289 17. Apr 09:56 meaning.dic
-rw-r--r-- 1 splunk splunk 7615 17. Apr 09:56 suffix_list_custom.dat
-rw-r--r-- 1 splunk splunk 7432 17. Apr 09:56 suffix_list_iana.dat
-rw-r--r-- 1 splunk splunk 157772 17. Apr 09:56 suffix_list_mozilla.dat
-rw-r--r-- 1 splunk splunk 2055 17. Apr 09:56 ut_bayesian.py
-rw-r--r-- 1 splunk splunk 2217 17. Apr 09:56 ut_countset.py
-rw-r--r-- 1 splunk splunk 1151 17. Apr 09:56 ut_levenshtein.py
-rw-r--r-- 1 splunk splunk 406 17. Apr 09:56 ut_log.py
-rw------- 1 splunk splunk 746 17. Apr 11:11 ut_log.pyc
-rw-r--r-- 1 splunk splunk 1802 17. Apr 09:56 ut_meaning.py
-rw-r--r-- 1 splunk splunk 1071 17. Apr 09:56 ut_parse_extended.py
-rw-r--r-- 1 splunk splunk 6199 17. Apr 09:56 ut_parse_lib.py
-rw------- 1 splunk splunk 6099 17. Apr 11:11 ut_parse_lib.pyc
-rw-r--r-- 1 splunk splunk 758 17. Apr 09:56 ut_parse_simple.py
-rw-r--r-- 1 splunk splunk 2405 17. Apr 09:56 ut_presets.py
-rw-r--r-- 1 splunk splunk 874 17. Apr 09:56 ut_shannon.py
-rw-r--r-- 1 splunk splunk 677 17. Apr 09:56 ut_suites.py
Using the correct syntax fixed my problem...
| makeresults
| eval url="http://sub.domain.com/folder/?test=ok", list="mozilla"
| `ut_parse_extended(url,list)`
Now all Fields are extracted from url.
_time,list,url,"ut_domain","ut_domain_without_tld","ut_fragment","ut_netloc","ut_params","ut_path","ut_port","ut_query","ut_scheme","ut_subdomain","ut_subdomain_count","ut_subdomain_level_1","ut_tld"
2019-04-23T11:45:40.000+0200,mozilla,"http://sub.domain.com/folder/?test=ok","domain.com",domain,None,"sub.domain.com",None,"/folder/",80,"test=ok",http,sub,1,sub,com