map_notable_fields in ES bug ?

agneticdk — Wed, 30 Jan 2019 09:22:43 GMT

Hi guys

I have this search:

| datamodel "Malware" "Malware_Attacks" search | `drop_dm_object_name(Malware_Attacks)` |`map_notable_fields` | search abc

It does NOT filter on the abc content.

If I do this:

| datamodel "Malware" "Malware_Attacks" search | `drop_dm_object_name(Malware_Attacks)` | search abc |`map_notable_fields`

it works.

I can see in the job inspector that the first search adds a "| noop" to the normalized search just before the "|search abc":

| search (tag=attack tag=malware (index=* OR index=_*) (index="estreamer" OR index="pan" OR index="trendmicro" OR index="wineventlog")) DIRECTIVES(READ_SUMMARY(allow_old_summaries="false" dmid="FAC94FC4-23AE-44F4-BA73-E65513A137DB_DM_Splunk_SA_CIM_Malware" name="Malware.Malware_Attacks" predicate="*" summariesonly="false"),REQUIRED_TAGS(intersect="t" tags="pci")) | eval action=if((isnull(action) OR (action == "")),"unknown",action), category=if((isnull(category) OR (category == "")),"unknown",category), date=strftime('_time',"%m-%d-%Y"), signature=if((isnull(signature) OR (signature == "")),"unknown",signature), dest=if((isnull(dest) OR (dest == "")),"unknown",dest), dest_nt_domain=if((isnull(dest_nt_domain) OR (dest_nt_domain == "")),"unknown",dest_nt_domain), user=if((isnull(user) OR (user == "")),"unknown",user), vendor_product=case(isnotnull(vendor_product),vendor_product,(((isnotnull(vendor) AND (vendor != "unknown")) AND isnotnull(product)) AND (product != "unknown")),((vendor . " ") . product),((isnotnull(vendor) AND (vendor != "unknown")) AND (isnull(product) OR (product == "unknown"))),(vendor . " unknown"),(((isnull(vendor) OR (vendor == "unknown")) AND isnotnull(product)) AND (product != "unknown")),("unknown " . product),isnotnull(sourcetype),sourcetype,true(),"unknown"), is_Allowed_Malware=if(searchmatch("action=\"allowed\""),1,0), is_not_Allowed_Malware=(1 - is_Allowed_Malware), is_Blocked_Malware=if(searchmatch("action=\"blocked\""),1,0), is_not_Blocked_Malware=(1 - is_Blocked_Malware), is_Deferred_Malware=if(searchmatch("action=\"deferred\""),1,0), is_not_Deferred_Malware=(1 - is_Deferred_Malware) | rename action as "Malware_Attacks.action", category as "Malware_Attacks.category", date as "Malware_Attacks.date", dest as "Malware_Attacks.dest", dest_bunit as "Malware_Attacks.dest_bunit", dest_category as "Malware_Attacks.dest_category", dest_nt_domain as "Malware_Attacks.dest_nt_domain", dest_priority as "Malware_Attacks.dest_priority", dest_requires_av as "Malware_Attacks.dest_requires_av", file_hash as "Malware_Attacks.file_hash", file_name as "Malware_Attacks.file_name", file_path as "Malware_Attacks.file_path", is_Allowed_Malware as "Malware_Attacks.is_Allowed_Malware", is_Blocked_Malware as "Malware_Attacks.is_Blocked_Malware", is_Deferred_Malware as "Malware_Attacks.is_Deferred_Malware", is_not_Allowed_Malware as "Malware_Attacks.is_not_Allowed_Malware", is_not_Blocked_Malware as "Malware_Attacks.is_not_Blocked_Malware", is_not_Deferred_Malware as "Malware_Attacks.is_not_Deferred_Malware", sender as "Malware_Attacks.sender", signature as "Malware_Attacks.signature", src as "Malware_Attacks.src", src_bunit as "Malware_Attacks.src_bunit", src_category as "Malware_Attacks.src_category", src_priority as "Malware_Attacks.src_priority", tag as "Malware_Attacks.tag", url as "Malware_Attacks.url", user as "Malware_Attacks.user", user_bunit as "Malware_Attacks.user_bunit", user_category as "Malware_Attacks.user_category", user_priority as "Malware_Attacks.user_priority", vendor_product as "Malware_Attacks.vendor_product" | fields + _time, host, source, sourcetype, "Malware_Attacks.dest_bunit", "Malware_Attacks.dest_category", "Malware_Attacks.dest_priority", "Malware_Attacks.dest_requires_av", "Malware_Attacks.file_hash", "Malware_Attacks.file_name", "Malware_Attacks.file_path", "Malware_Attacks.sender", "Malware_Attacks.src", "Malware_Attacks.src_bunit", "Malware_Attacks.src_category", "Malware_Attacks.src_priority", "Malware_Attacks.tag", "Malware_Attacks.url", "Malware_Attacks.user_bunit", "Malware_Attacks.user_category", "Malware_Attacks.user_priority", "Malware_Attacks.action", "Malware_Attacks.category", "Malware_Attacks.date", "Malware_Attacks.signature", "Malware_Attacks.dest", "Malware_Attacks.dest_nt_domain", "Malware_Attacks.user", "Malware_Attacks.vendor_product", "Malware_Attacks.is_Allowed_Malware", "Malware_Attacks.is_not_Allowed_Malware", "Malware_Attacks.is_Blocked_Malware", "Malware_Attacks.is_not_Blocked_Malware", "Malware_Attacks.is_Deferred_Malware", "Malware_Attacks.is_not_Deferred_Malware" | rename "Malware_Attacks.*" as "*" | noop| search abc

Do any of you know why this is the behavior, and why the macro in itself changes Splunks behavior.
Also if I expand the macros, my search works fine, and the noop is not added in the expanded search.
This is also the behavior in the search app, and not just the ES search.

Re: map_notable_fields in ES bug ?

DMohn — Wed, 30 Jan 2019 09:49:30 GMT

As far as I know, the map_notable_fields makro is a legacy component, which is no longer neccessary in current versions of Enterprise Security. This is why in the macro definition it is defined as noop, which means 'no operation' - so basically it does nothing. noopcan be used for debug purposes though, you can check https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Noop for reference

You can remove the macro completely, it should not make any difference. Why it breaks your search if you use it before the filter part is something I do not completely understand. Maybe it mingles around with the search optimization (which is one of the noop debug features). Have you turned off the search optimization in your limits.conf? This maybe could explain the different behavior.

Re: map_notable_fields in ES bug ?

agneticdk — Wed, 30 Jan 2019 10:07:09 GMT

Ahh yes - I see it now. Thank you

topic map_notable_fields in ES bug ? in Splunk Enterprise Security

map_notable_fields in ES bug ?

Re: map_notable_fields in ES bug ?

Re: map_notable_fields in ES bug ?