Unable to filter out lookup table values

[browncardigan]

I'm trying to filter out false-positive domains in a search of DNS events by using NOT on the ut_domain field of the lookup table. The search runs but provides no results, despite there being events that should've been returned in the results.

sourcetype=infoblox:dns record_type!=SRV record_type!=PTR record_type!=SOA
| eval list="mozilla"
| `ut_parse(query, list)`
| search ut_domain NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
| `ut_shannon(ut_subdomain)`
| search ut_shannon >= 4.0
| stats count by ut_netloc ut_shannon
| sort -ut_shannon, -count

The search works fine until I try to filter out the domains.

I've tried multiple iterations including re-naming the lookup file's column name from ut_domain to domain and removing the ut_shannon >= 4.0 component, but nothing's worked:

| search NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
| search ut_domain NOT [|inputlookup FP_entropy_domains.csv | fields domain]
| search NOT [|inputlookup FP_entropy_domains.csv | fields domain]  

Splunk indicates that it's successfully looking up the csv, and the comparison to the domain/ut_domain column in the lookup table is happening - I just don't get any results at the end of it.

Every answer I find on Splunk Answers indicates it should work with the simple |search NOT [|inputlookup | fields ] and I can't see what I'm doing wrong in this case.

[to4kawa]

Hi,

[|inputlookup FP_entropy_domains.csv | fields ut_domain]

returns (ut_domain=xx OR ut_domain=yy OR ....)

sourcetype=infoblox:dns 　

What fields are extracted from this result?
If it is domain, you need to change the field name.

[|inputlookup FP_entropy_domains.csv | fields ut_domain|rename ut_domain as domain]

[browncardigan]

 sourcetype=infoblox:dns record_type!=SRV record_type!=PTR record_type!=SOA
 | eval list="mozilla"
 | `ut_parse(query, list)`
 | search ut_domain NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
 | `ut_shannon(ut_subdomain)`
 | search ut_shannon >= 4.0
 | stats count by ut_netloc ut_shannon
 | sort -ut_shannon, -count

Line 1 pulls all non SRV, PTR, and SOA DNS records from the infoblox:dns sourcetype
Lines 2 - 3 use the URLToolbox App to parse the DNS queries using a list of Mozilla-recognised domains. This parses the domains out in the field "ut_domain".

I've tried putting the false-positive domains in the csv under both ut_domain and domain, and neither have worked. Using the rename command in the sub search wouldn't achieve anything different as far as I'm aware.

[to4kawa]

I think that the command is different when I see the following blog.
UT_parsing Domains Like House Slytherin
Random Words on Entropy and DNS

ut_parse → ut_parse_extended

| search  NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]

[browncardigan]

ut_parse and ut_parse_extended are the same thing according to the utbox documentation:

ut_parse(url, list) or ut_parse_extended(url, list):
Extended version of ut_parse_simple() which uses a list to extract the following
fields: ut_port, ut_domain, ut_tld, ut_domain_without_tld, ut_subdomain,
ut_subdomain_count and ut_subdomain_parts.