All Apps and Add-ons
Highlighted

Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Engager

After upgrading the Palo Alto Networks App for Splunk from version 5.4 to 6, I am getting the following error message from each member of the index cluster (3 member cluster).

[idx1] The lookup table 'minemeldfeedslookup' does not exist. It is referenced by configuration 'pan:threat'.
[idx1] The lookup table 'minemeldfeeds
lookup' does not exist. It is referenced by configuration 'pan:traffic'.

I have verified the latest SplunkforPaloAltoNetworks and SplunkTApaloalto are both installed on each indexer as well as the search head. How do I go about correcting this error?,I am getting the following errors from our Indexers (setup up in a 3 member cluster) after upgrading to version 6.0 from version 5.4.0.

[idx] The lookup table 'minemeldfeedslookup' does not exist. It is referenced by configuration 'pan:threat'.
[idx] The lookup table 'minemeldfeeds
lookup' does not exist. It is referenced by configuration 'pan:traffic'.

How do I correct these errors?

Highlighted

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Champion

That means that you have an automatic lookup configured for those sourcetypes. But when the lookup referenced by the automatic lookup doesn't exist. You might want to start by seeing if that automatic lookup is actually defined in the latest version of the app or something that is left over.

But essentially, you either need to disable/remove the automatic lookup if it's not needed. OR find out why the lookup up it references is missing. For example, maybe a scheduled search creates that lookup and it hasn't run yet or hasn't produced any results

Settings -> Lookups -> Automatic Lookups (these are in props.conf)
This is where you define which fields to lookup, which to spit out and for which sourcetype

Settings -> Lookups -> Lookup definitions (these are in transforms.conf)
This is likely referenced by the automatic lookup and contains the lookup file info, and settings for lookup (like case sensitivity, wildcard matching, etc)

Settings -> Lookups -> Lookup table files (these are the actual files under /lookups)
This will show the path to the actual lookup file mentioned the lookup reference.

Side note: The reason the messages are coming from the indexers is not that the lookup is missing from the indexers app (under slaveapps) but that the search bundle pushed down to the indexers from the search head is missing the lookup. So one other possibility is that the lookup is blacklisted from the search bundle (distsearch.conf), but that's probably not too likely in this case.

0 Karma
Highlighted

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Splunk Employee
Splunk Employee

You must enable replication for the kvstore collection [minemeldfeeds]

KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

Events of sourcetype pan:threat and pan:traffic have automatic lookups configured in props.conf / transforms.conf so the automatic lookup against the minemeldfeeds collection is running on the indexers but the minemeldfeeds kvstore collection doesn't exist there because it is not being replicated from the SH to the indexers by default.

ie:
/opt/splunk/etc/apps/SplunkTApaloalto/default/props.conf

LOOKUP-minemeldfeedsdestlookup = minemeldfeeds_lookup indicator AS destip OUTPUT value.autofocustags AS destautofocustags

/opt/splunk/etc/apps/SplunkTApaloalto/default/transforms.conf

'[minemeldfeedslookup]'
collection = minemeldfeeds
external
type = kvstore
fieldslist = indicator, splunksource, value.confidence, value.firstseen, value.lastseen, value.sharelevel, value.sources, value.type, value.autofocustags, value.autofocusbenign, value.autofocusgrayware, value.autofocusmalware, value.autofocusnummatchingartifacts, value

/opt/splunk/etc/apps/SplunkTApaloalto/default/collections.conf

[minemeldfeeds]
field.indicator = string
field.splunk_source = string
field.value = string

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

On your search heads create a file called collections.conf in $SPLUNK_HOME/etc/system/local/ (or within an app)
[minemeldfeeds]
replicate = true

Restart Splunk to apply your changes.
Once the next bundle is replicated to your indexers the kvstore collection definition will exist at the indexers and you should see this error message go away the next search you run.

If your indexers are running a version of Splunk Enterprise that is older than 6.3, attempts to run an automatic lookup fail with a "lookup does not exist" error. You must upgrade your indexers to 6.3 or later to use this functionality.

you can learn more about automatic lookups and kvstore collection replication here:
https://docs.splunk.com/Documentation/Splunk/7.0.0/Knowledge/Makeyourlookupautomatic

View solution in original post

Highlighted

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Engager

Thank you, that cleared it up for me.

0 Karma
Highlighted

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Path Finder

Thank you @rphillips - that was helpful. Quick clarification though..

You had said

On your search heads create a file called collections.conf in $SPLUNK_HOME/etc/system/local/ (or within an app)

Did you mean indexer (cluster)? Those error messages didn't go away for me until I did the above on the index cluster (That stanza in the collections.conf is also on my SHC).

Thanks!

0 Karma
Highlighted

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Splunk Employee
Splunk Employee

@rpquinlan the configuration in collections.conf should be placed on the SHs. Upon the next knowledge bundle replication (SH-->Indexers) the kvstore collection will be replicated to the indexers. Its likely that you were still seeing the errors because a new knowledge bundle hadn't yet been sent to the indexers.

0 Karma
Highlighted

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Builder

Did this suggestion fix the problem?

0 Karma
Highlighted

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Communicator

I was able to resolve this issue, here is the process I have followed

I have updated (replicate = true) parameter in collections.conf file on these instances (Cluster Search Heads Members as well as on Enterprise Search Head Instance).

Make sure you copy the collections.conf under local directory and make changes, /opt/splunk/etc/apps/SplunkTApaloalto/local/collections.conf

[minemeldfeeds]
replicate = true
field.indicator = string
field.splunk_source = string
field.value = string

Stanza should like above.

Push these changes on the Cluster Search Head through deployer such that you have similar changes across cluster.

Run the below command as per your environment
/opt/splunk/bin/splunk apply shcluster-bundle target {cluster search head captain} -auth

If the enterprise search head is managed by the deployment server then try to follow the below process

Edit the add-on collections.conf under local directory by making a local copy and reload the deployment server

/opt/splunk/bin/splunk reload deploy-server

This should fix the issue, you can also ensure whether the knowledge bundle is applied successfully or not in two ways.

One, try to search data through UI (Enterprise Search Head).

Second, login into your indexer server and cd this path /opt/splunk/etc/var/run/searchpeers/. Here you can see the Knowledge bundles that are applied and check if the latest bundle applied has the change that you have made which got either pushed through the deployment or deployer.

0 Karma
Highlighted

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Path Finder

@vgollapudi Hi,

I am seeing same error after upgrading my splunk. Can you post your advise on this?
The error I am seeing on the serach head is
Could not load lookup=LOOKUP-minemeldfeedsdestlookup
Could not load lookup=LOOKUP-minemeldfeedssrclookup.
Search peer errors are same as above.

Note: search head is running 7.2 and search peers are running 6.6.6

0 Karma
Highlighted

Re: Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

Communicator

Hello,

Did you follow the steps listed by me or rphilips? If you haven't do follow them, I did share the steps which helped me to resolve the error.

The Search Head version of our environment is 6.6.3. I'm unsure what has changed in 7.2 version.

0 Karma