topic Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0 in All Apps and Add-ons

Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

veddersg — Tue, 29 Sep 2020 16:44:33 GMT

After upgrading the Palo Alto Networks App for Splunk from version 5.4 to 6, I am getting the following error message from each member of the index cluster (3 member cluster).

[idx1] The lookup table 'minemeldfeeds_lookup' does not exist. It is referenced by configuration 'pan:threat'.
[idx1] The lookup table 'minemeldfeeds_lookup' does not exist. It is referenced by configuration 'pan:traffic'.

I have verified the latest SplunkforPaloAltoNetworks and Splunk_TA_paloalto are both installed on each indexer as well as the search head. How do I go about correcting this error?,I am getting the following errors from our Indexers (setup up in a 3 member cluster) after upgrading to version 6.0 from version 5.4.0.

[idx] The lookup table 'minemeldfeeds_lookup' does not exist. It is referenced by configuration 'pan:threat'.
[idx] The lookup table 'minemeldfeeds_lookup' does not exist. It is referenced by configuration 'pan:traffic'.

How do I correct these errors?

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

maciep — Wed, 15 Nov 2017 23:01:43 GMT

That means that you have an automatic lookup configured for those sourcetypes. But when the lookup referenced by the automatic lookup doesn't exist. You might want to start by seeing if that automatic lookup is actually defined in the latest version of the app or something that is left over.

But essentially, you either need to disable/remove the automatic lookup if it's not needed. OR find out why the lookup up it references is missing. For example, maybe a scheduled search creates that lookup and it hasn't run yet or hasn't produced any results

Settings -> Lookups -> Automatic Lookups (these are in props.conf)
This is where you define which fields to lookup, which to spit out and for which sourcetype

Settings -> Lookups -> Lookup definitions (these are in transforms.conf)
This is likely referenced by the automatic lookup and contains the lookup file info, and settings for lookup (like case sensitivity, wildcard matching, etc)

Settings -> Lookups -> Lookup table files (these are the actual files under /lookups)
This will show the path to the actual lookup file mentioned the lookup reference.

Side note: The reason the messages are coming from the indexers is not that the lookup is missing from the indexers app (under slaveapps) but that the search bundle pushed down to the indexers from the search head is missing the lookup. So one other possibility is that the lookup is blacklisted from the search bundle (distsearch.conf), but that's probably not too likely in this case.

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

rphillips_splk — Tue, 29 Sep 2020 16:44:46 GMT

You must enable replication for the kvstore collection [minemeldfeeds]

KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

Events of sourcetype pan:threat and pan:traffic have automatic lookups configured in props.conf / transforms.conf so the automatic lookup against the minemeldfeeds collection is running on the indexers but the minemeldfeeds kvstore collection doesn't exist there because it is not being replicated from the SH to the indexers by default.

ie:
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/props.conf

LOOKUP-minemeldfeeds_dest_lookup = minemeldfeeds_lookup indicator AS dest_ip OUTPUT value.autofocus_tags AS dest_autofocus_tags

/opt/splunk/etc/apps/Splunk_TA_paloalto/default/transforms.conf

'[minemeldfeeds_lookup]'
collection = minemeldfeeds
external_type = kvstore
fields_list = indicator, splunk_source, value.confidence, value.first_seen, value.last_seen, value.share_level, value.sources, value.type, value.autofocus_tags, value.autofocus_benign, value.autofocus_grayware, value.autofocus_malware, value.autofocus_num_matching_artifacts, value

/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf

[minemeldfeeds]
field.indicator = string
field.splunk_source = string
field.value = string

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

On your search heads create a file called collections.conf in $SPLUNK_HOME/etc/system/local/ (or within an app)
[minemeldfeeds]
replicate = true

Restart Splunk to apply your changes.
Once the next bundle is replicated to your indexers the kvstore collection definition will exist at the indexers and you should see this error message go away the next search you run.

If your indexers are running a version of Splunk Enterprise that is older than 6.3, attempts to run an automatic lookup fail with a "lookup does not exist" error. You must upgrade your indexers to 6.3 or later to use this functionality.

you can learn more about automatic lookups and kvstore collection replication here:
https://docs.splunk.com/Documentation/Splunk/7.0.0/Knowledge/Makeyourlookupautomatic

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

panguy — Thu, 16 Nov 2017 01:10:19 GMT

Hi could you help me understand where these error messages are showing up? Are they showing up under 'Messages' from the navigation?

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

rphillips_splk — Thu, 16 Nov 2017 01:18:43 GMT

The message will show up after you run a search if you haven't replicated the kvstore collection

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

veddersg — Thu, 16 Nov 2017 02:23:18 GMT

Thank you, that cleared it up for me.

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

rpquinlan — Thu, 16 Nov 2017 20:34:19 GMT

Thank you @rphillips - that was helpful. Quick clarification though..

You had said

On your search heads create a file called collections.conf in $SPLUNK_HOME/etc/system/local/ (or within an app)

Did you mean indexer (cluster)? Those error messages didn't go away for me until I did the above on the index cluster (That stanza in the collections.conf is also on my SHC).

Thanks!

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

rphillips_splk — Fri, 17 Nov 2017 19:45:10 GMT

@rpquinlan the configuration in collections.conf should be placed on the SHs. Upon the next knowledge bundle replication (SH-->Indexers) the kvstore collection will be replicated to the indexers. Its likely that you were still seeing the errors because a new knowledge bundle hadn't yet been sent to the indexers.

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

btorresgil — Fri, 17 Nov 2017 22:17:31 GMT

Did this suggestion fix the problem?

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

vgollapudi — Tue, 29 Sep 2020 16:50:30 GMT

I was able to resolve this issue, here is the process I have followed

I have updated (replicate = true) parameter in collections.conf file on these instances (Cluster Search Heads Members as well as on Enterprise Search Head Instance).

Make sure you copy the collections.conf under local directory and make changes, /opt/splunk/etc/apps/Splunk_TA_paloalto/local/collections.conf

[minemeldfeeds]
replicate = true
field.indicator = string
field.splunk_source = string
field.value = string

Stanza should like above.

Push these changes on the Cluster Search Head through deployer such that you have similar changes across cluster.

Run the below command as per your environment
/opt/splunk/bin/splunk apply shcluster-bundle target {cluster search head captain} -auth

If the enterprise search head is managed by the deployment server then try to follow the below process

Edit the add-on collections.conf under local directory by making a local copy and reload the deployment server

/opt/splunk/bin/splunk reload deploy-server

This should fix the issue, you can also ensure whether the knowledge bundle is applied successfully or not in two ways.

One, try to search data through UI (Enterprise Search Head).

Second, login into your indexer server and cd this path /opt/splunk/etc/var/run/searchpeers/. Here you can see the Knowledge bundles that are applied and check if the latest bundle applied has the change that you have made which got either pushed through the deployment or deployer.

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

weicai88 — Tue, 29 Sep 2020 17:40:15 GMT

rphillips:

I got the same error, and following your instructions, checked my settings with btool. Here's what's on my search heads:

/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf [minemeldfeeds]
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf enforceTypes = false
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf field.indicator = string
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf field.splunk_source = string
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf field.value = string
/opt/splunk/etc/system/default/collections.conf profilingEnabled = false
/opt/splunk/etc/system/default/collections.conf profilingThresholdMs = 1000
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf replicate = true
/opt/splunk/etc/system/default/collections.conf replication_dump_maximum_file_size = 10240
/opt/splunk/etc/system/default/collections.conf replication_dump_strategy = auto
/opt/splunk/etc/system/default/collections.conf type = undefined

As you can see, replicate = true. Why do i still see the error?

Thanks!
Wei

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

btorresgil — Tue, 16 Jan 2018 20:50:12 GMT

@weicai88: Have you installed the App and Add-on 6.0.1 or higher on all Searchheads, Indexers, and Heavy Forwarders?

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

weicai88 — Wed, 17 Jan 2018 15:31:24 GMT

I only installed the Add-on 6.0.2, not the App. And yes, it is on both Search Heads and Indexers. It's odd that I have two Search Head clusters, but the errors only show up on one of them although the Add-on was installed on both.

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

graju89 — Tue, 29 Sep 2020 21:46:23 GMT

@vgollapudi Hi,

I am seeing same error after upgrading my splunk. Can you post your advise on this?
The error I am seeing on the serach head is
Could not load lookup=LOOKUP-minemeldfeeds_dest_lookup
Could not load lookup=LOOKUP-minemeldfeeds_src_lookup.
Search peer errors are same as above.

Note: search head is running 7.2 and search peers are running 6.6.6

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

vgollapudi — Tue, 30 Oct 2018 00:35:30 GMT

Hello,

Did you follow the steps listed by me or rphilips? If you haven't do follow them, I did share the steps which helped me to resolve the error.

The Search Head version of our environment is 6.6.3. I'm unsure what has changed in 7.2 version.

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

graju89 — Tue, 30 Oct 2018 13:18:38 GMT

@vgollapudi Yes, I followed it. But still I am seeing the error.

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

btorresgil — Tue, 30 Oct 2018 15:36:42 GMT

Replication is on by default in the latest Add-on. Try re-installing the Add-on, it sounds like something is broken after the upgrade.

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

graju89 — Tue, 30 Oct 2018 15:46:30 GMT

@btorresgil I tried re-installing too. Stil no luck

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

illao — Fri, 01 Mar 2019 07:10:46 GMT

@graju89 I've same issue just like yours and finally I found that the KV store is disabled on my search head. Once kv store enable and the Knowledge bundles is replicated to search peer, the error is gone.

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

hketer — Thu, 12 Nov 2020 18:17:09 GMT

Hi All,

I've ran a search and got these errors

I've checked the lookup and it's empty

Is there a way to recover it?

Thanks,

Hen