All Apps and Add-ons

URL Toolbox problem with ut_parse_extended / ut_parse_extended_lookup

ssteinmann
Explorer

Hi all,

i'm using URL Toolbox for analysing url entropy, shannon and meaning scoring is working also simple parse of url. I now want to extract the main url to match against some TI List but the ut_parse_extended(url, list) macro does not work. I'm using latest URl Toolbox Version 1.6 and Splunk 7.2.x.

This is my example Query:

| makeresults 
| eval url="www.domain.com"
| lookup ut_parse_simple_lookup url as url
| lookup ut_parse_extended_lookup url as url list as mozilla
| `ut_shannon(url)` 
| `ut_meaning(url)`

This does not include extractions from extended parse...any idea?

_time   url ut_fragment ut_meaning_ratio    ut_netloc   ut_params   ut_path ut_query    ut_scheme   ut_shannon
17.04.2019 11:49    www.domain.com  None    0.35714285714285715 www.domain.com  None    None    None    None    3.6978458230844127

all phyton scripts are there:

ll /opt/splunk/etc/apps/utbox/bin/
-rw-r--r-- 1 splunk splunk 359422 17. Apr 09:56 bayesian_bad.dic
-rw-r--r-- 1 splunk splunk 269354 17. Apr 09:56 bayesian_good.dic
-rw-r--r-- 1 splunk splunk  37289 17. Apr 09:56 meaning.dic
-rw-r--r-- 1 splunk splunk   7615 17. Apr 09:56 suffix_list_custom.dat
-rw-r--r-- 1 splunk splunk   7432 17. Apr 09:56 suffix_list_iana.dat
-rw-r--r-- 1 splunk splunk 157772 17. Apr 09:56 suffix_list_mozilla.dat
-rw-r--r-- 1 splunk splunk   2055 17. Apr 09:56 ut_bayesian.py
-rw-r--r-- 1 splunk splunk   2217 17. Apr 09:56 ut_countset.py
-rw-r--r-- 1 splunk splunk   1151 17. Apr 09:56 ut_levenshtein.py
-rw-r--r-- 1 splunk splunk    406 17. Apr 09:56 ut_log.py
-rw------- 1 splunk splunk    746 17. Apr 11:11 ut_log.pyc
-rw-r--r-- 1 splunk splunk   1802 17. Apr 09:56 ut_meaning.py
-rw-r--r-- 1 splunk splunk   1071 17. Apr 09:56 ut_parse_extended.py
-rw-r--r-- 1 splunk splunk   6199 17. Apr 09:56 ut_parse_lib.py
-rw------- 1 splunk splunk   6099 17. Apr 11:11 ut_parse_lib.pyc
-rw-r--r-- 1 splunk splunk    758 17. Apr 09:56 ut_parse_simple.py
-rw-r--r-- 1 splunk splunk   2405 17. Apr 09:56 ut_presets.py
-rw-r--r-- 1 splunk splunk    874 17. Apr 09:56 ut_shannon.py
-rw-r--r-- 1 splunk splunk    677 17. Apr 09:56 ut_suites.py
Tags (1)
0 Karma

ssteinmann
Explorer

Using the correct syntax fixed my problem...

| makeresults 
| eval url="http://sub.domain.com/folder/?test=ok", list="mozilla"
| `ut_parse_extended(url,list)`

Now all Fields are extracted from url.

_time,list,url,"ut_domain","ut_domain_without_tld","ut_fragment","ut_netloc","ut_params","ut_path","ut_port","ut_query","ut_scheme","ut_subdomain","ut_subdomain_count","ut_subdomain_level_1","ut_tld"
2019-04-23T11:45:40.000+0200,mozilla,"http://sub.domain.com/folder/?test=ok","domain.com",domain,None,"sub.domain.com",None,"/folder/",80,"test=ok",http,sub,1,sub,com
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...