Splunk Enterprise Security new log ERRORs after upgrading enterprise to 7.3.3 and ESS from 5.0.1 to 5.3.1

[kmarciniak]

I need to determine the significance of these errors before giving the green light to upgrade production. These are all related to after upgrading enterprise from 7.0.5 to 7.3.3 and Splunk Enterprise security from 5.0.1 to 5.3.1.

ERROR 2019-12-10 10:01:52.639 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T http_intel * where * * * '.

ERROR 2019-12-10 10:01:52.649 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T email_intel * where * * '.

ERROR 2019-12-10 10:01:52.670 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T service_intel * where * '.

ERROR 2019-12-10 10:01:52.679 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T registry_intel * where * '.

ERROR 2019-12-10 10:01:52.689 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T process_intel * where * *'.

ERROR 2019-12-10 10:01:52.708 security01-dev SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T user_intel * where * * * '.

ERROR 2019-12-10 10:01:52.718 security01-dev SearchParser Missing a search command before '*'. Error at position '2' of search query '| * inputlookup append=T certificate_intel * where'.

If I run a grep on all files with these variable such as http_intel, email_intel, service_intel, the results show them in ESS DA dashboards. We never saw this error prior to upgrading to 7.3.3

Below is a sample, all others are about identical.

ERROR 2019-12-10 04:01:39.131 security SearchParser Missing a search command before '*'. Error at position '2' of search query '| * inputlookup append=T http_intel * where * * *

grep on http_intel found the below dashboard search

DA-ESS-ThreatIntelligence/default/data/ui/views/threat_artifacts.xml:432: | $tab_threat$ inputlookup append=T process_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("process_intel") | eval ip=mvappend(src, dest), domain=mvappend(src, dest) | inputlookup append=T certificate_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("certificate_intel") | mvexpand certificate_serial | get_certificate_serial | eventstats values(certificate_serial) as certificate_serial,values(certificate_serial_clean) as certificate_serial_clean,values(certificate_serial_dec) as certificate_serial_dec by _key | dedup _key,threat_collection | inputlookup append=T email_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("email_intel") | inputlookup append=T ip_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("ip_intel") | inputlookup append=T http_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | fillnull value=0 updated,disabled | set_threat_collection_name("http_intel") | get_threat_attribution(threat_key) | search $threat_id_filter$ | eval ip=coalesce(embedded_ip,ip), domain=coalesce(embedded_domain,domain), file_hash=coalesce(certificate_file_hash,file_hash), src_user=coalesce(certificate_subject_email,src_user), actual_src_user=coalesce(certificate_issuer_email,actual_src_user), file_name=coalesce(process_file_name,file_name), file_path=coalesce(process_file_path,file_path) | mvappend_field(url, http_referrer) | fillnull value="" threat_collection, source_type, threat_group, threat_category, malware_alias| stats dc(ip) as ip_count, dc(domain) as domain_count, dc(url) as url_count, dc(http_user_agent) as http_user_agent_count, dc(header) as header_count by threat_collection, source_type, threat_group, threat_category, malware_alias | addtotals fieldname=http_count http_user_agent_count, header_count | addtotals fieldname=total ip_count, domain_count, url_count, http_count | where total > 0 | fields threat_collection, source_type, ip_count, domain_count, url_count, http_count, total, threat_group, threat_category, malware_alias | sort - total

[kmarciniak]

received response from support
Thanks for contacting Splunk Support. This is known issue with the logging. The message is benign as it deals with how we log search operations during parsing and wildcard substitution phase of the search.

There is a bug SPL-170703 open in regards to have better logging in regards to this.

[jlarsonq]

I am pretty sure the errors are benign, it is coming from an internal Splunk ES process that it verifying the dashboards and the search parser that is checking is putting in the dummy "*" and then erroring out from it. It isn't actually ever trying to run that search.

[xavierashe]

I hope you're right. My users are having issues with the Incident Response dashboard and we're try to locate the problem. If we come up with anything, I'll come back an update this thread.