Ended up altering the threat_gen search for URL matches to make the url field an mvfield with the three most likely url protocol prefixes. The only line I added to the stock code was | eval url=mvappend("http://".url, "https://".url, "ftp://".url) , shown in line below. This allows the lookups and the domain extraction to function properly, and the analyst is able to review the original log to see the actual protocol used. If you have Zscaler and need a cleaner solution, you can probably add "Web.transport" to the |tstats ... by clause and actually build the correct value with an eval case statement, but I went with the mvfield option for its simplicity.
| `tstats` values(sourcetype) as sourcetype,values(Web.src),values(Web.dest) from datamodel=Web.Web by Web.http_referrer
| eval url='Web.http_referrer'
| eval threat_match_field="http_referrer"
| `tstats` append=true values(sourcetype) as sourcetype,values(Web.src),values(Web.dest) from datamodel=Web.Web by Web.url
| eval url=if(isnull(url),'Web.url',url)
| eval threat_match_field=if(isnull(threat_match_field),"url",threat_match_field)
| stats values(sourcetype) as sourcetype,values(Web.src) as src,values(Web.dest) as dest by url,threat_match_field
| eval url=mvappend("http://".url, "https://".url, "ftp://".url)
| extract domain_from_url
| `threatintel_url_lookup(url)`
| `threatintel_domain_lookup(url_domain)`
| search threat_collection_key=*
| `mvtruncate(src)`
| `mvtruncate(dest)`
| `zipexpand_threat_matches`
... View more