Splunk Enterprise Security

Splunk Enterprise Security new log ERRORs after upgrading enterprise to 7.3.3 and ESS from 5.0.1 to 5.3.1

kmarciniak
Path Finder

I need to determine the significance of these errors before giving the green light to upgrade production. These are all related to after upgrading enterprise from 7.0.5 to 7.3.3 and Splunk Enterprise security from 5.0.1 to 5.3.1.

ERROR 2019-12-10 10:01:52.639 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T http_intel * where * * * '.

ERROR 2019-12-10 10:01:52.649 security SearchParser Missing a search command before '
'. Error at position '2' of search query '| * inputlookup append=T email_intel * where * * '.

ERROR 2019-12-10 10:01:52.670 security SearchParser Missing a search command before '
'. Error at position '2' of search query '| * inputlookup append=T service_intel * where * '.

ERROR 2019-12-10 10:01:52.679 security SearchParser Missing a search command before '
'. Error at position '2' of search query '| * inputlookup append=T registry_intel * where * '.

ERROR 2019-12-10 10:01:52.689 security SearchParser Missing a search command before ''. Error at position '2' of search query '| * inputlookup append=T process_intel * where * *'.

ERROR 2019-12-10 10:01:52.708 security01-dev SearchParser Missing a search command before '
'. Error at position '2' of search query '| * inputlookup append=T user_intel * where * * * '.

ERROR 2019-12-10 10:01:52.718 security01-dev SearchParser Missing a search command before '*'. Error at position '2' of search query '| * inputlookup append=T certificate_intel * where'.

If I run a grep on all files with these variable such as http_intel, email_intel, service_intel, the results show them in ESS DA dashboards. We never saw this error prior to upgrading to 7.3.3

Below is a sample, all others are about identical.

ERROR 2019-12-10 04:01:39.131 security SearchParser Missing a search command before '*'. Error at position '2' of search query '| * inputlookup append=T http_intel * where * * *

grep on http_intel found the below dashboard search

DA-ESS-ThreatIntelligence/default/data/ui/views/threat_artifacts.xml:432: | $tab_threat$ inputlookup append=T process_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("process_intel") | eval ip=mvappend(src, dest), domain=mvappend(src, dest) | inputlookup append=T certificate_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("certificate_intel") | mvexpand certificate_serial | get_certificate_serial | eventstats values(certificate_serial) as certificate_serial,values(certificate_serial_clean) as certificate_serial_clean,values(certificate_serial_dec) as certificate_serial_dec by _key | dedup _key,threat_collection | inputlookup append=T email_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("email_intel") | inputlookup append=T ip_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | set_threat_collection_name("ip_intel") | inputlookup append=T http_intel $max$ where * $network_filter$ $file_filter$ $registry_filter$ $service_filter$ $user_filter$ $process_filter$ $certificate_filter$ $email_filter$ | fillnull value=0 updated,disabled | set_threat_collection_name("http_intel") | get_threat_attribution(threat_key) | search $threat_id_filter$ | eval ip=coalesce(embedded_ip,ip), domain=coalesce(embedded_domain,domain), file_hash=coalesce(certificate_file_hash,file_hash), src_user=coalesce(certificate_subject_email,src_user), actual_src_user=coalesce(certificate_issuer_email,actual_src_user), file_name=coalesce(process_file_name,file_name), file_path=coalesce(process_file_path,file_path) | mvappend_field(url, http_referrer) | fillnull value="" threat_collection, source_type, threat_group, threat_category, malware_alias| stats dc(ip) as ip_count, dc(domain) as domain_count, dc(url) as url_count, dc(http_user_agent) as http_user_agent_count, dc(header) as header_count by threat_collection, source_type, threat_group, threat_category, malware_alias | addtotals fieldname=http_count http_user_agent_count, header_count | addtotals fieldname=total ip_count, domain_count, url_count, http_count | where total > 0 | fields threat_collection, source_type, ip_count, domain_count, url_count, http_count, total, threat_group, threat_category, malware_alias | sort - total

0 Karma

kmarciniak
Path Finder

received response from support
Thanks for contacting Splunk Support. This is known issue with the logging. The message is benign as it deals with how we log search operations during parsing and wildcard substitution phase of the search.

There is a bug SPL-170703 open in regards to have better logging in regards to this.

jlarsonq
Explorer

I am pretty sure the errors are benign, it is coming from an internal Splunk ES process that it verifying the dashboards and the search parser that is checking is putting in the dummy "*" and then erroring out from it. It isn't actually ever trying to run that search.

0 Karma

xavierashe
Contributor

I hope you're right. My users are having issues with the Incident Response dashboard and we're try to locate the problem. If we come up with anything, I'll come back an update this thread.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...