All Apps and Add-ons

Upgrading to Palo Alto Networks App for Splunk version 5.0 from 4.x

dxmiller
Explorer

We just recently followed the upgrade procedures as outlined in the documentation and are encountering for following errors when performing a search which looks to be happening for all of our indexers:

Could not find all of the specified destination fields in the lookup table.' for conf 'pan:hipmatch' and lookup table 'pan_vendor_info_lookup'.

Could not find all of the specified destination fields in the lookup table.' for conf 'pan:system' and lookup table 'pan_vendor_info_lookup'.

Could not find all of the specified destination fields in the lookup table.' for conf 'pan:threat' and lookup table 'pan_vendor_info_lookup'.

Any thoughts on how to get this resolved?

0 Karma

dxmiller
Explorer

During the upgrade process, I proceeded with deleting the entire lookup directory from the app as documented as you can see from the output of the ls command below. I'm now starting to think that this may have to do with the TA on the indexers and light forwarder as opposed to the actual app server.

 ls /opt/splunk/etc/apps/SplunkforPaloAltoNetworks/
appserver  default.old.20150827-092501  install  metadata
bin        default.old.20151123-110714  LICENSE  README.md
default    docs                         local    static
0 Karma

kchamplin_splun
Splunk Employee
Splunk Employee

My guess is that the old lookup file is still in the app directory. If you look at the lookup definition in the TA's (Splunk_TA_paloalto) props.conf there is an additional field compared to the definition int he props.conf in the 4.x version of the app.

Old Version of the lookup (4.4):
LOOKUP-vendor_info_for_pan_config = pan_vendor_info_lookup sourcetype OUTPUT vendor,product

New version of the lookup (inside Splunk_TA_paloalto):
LOOKUP-vendor_info_for_pan_config = pan_vendor_info_lookup sourcetype OUTPUT vendor,product,vendor_product

This is consistent with the three sourcetypes you mention above.

Can you give the below a shot and see if that resolves the issue?

http://pansplunk.readthedocs.org/en/latest/upgrade.html#upgrade-to-app-version-5-0
"Delete any lookups in the App that you did not create. If you did not create any lookups in the App directory, then you can safely delete the entire lookup directory from the App. The path to the lookup directory is $SPLUNK_HOME/etc/apps/SplunkforPaloAltoNetworks/lookups"

0 Karma

dxmiller
Explorer

I have verified that the lookups directory resides on the indexers and light forwarder. Should the directory be removed from there as well? If so, I'll have to pull it from under:
$SPLUNK_HOME/etc/master-apps/Splunk_TA_paloalto and push out a new configuration bundle to our indexers. Thanks in advance!

0 Karma

kchamplin_splun
Splunk Employee
Splunk Employee

Yes, please try removing the old directory from the indexers (I don't think it should matter if it's sitting on the forwarder) - just in case please just back up the existing directory first and push out the new bits. Just to be clear, the old lookups are installed where on the indexers?

0 Karma

dxmiller
Explorer

Correct. The old lookups are out on our indexers. Moved directories to /tmp/old_lookups and restarted Splunk on all of the indexers. Still no luck and here is the latest round of errors that we're getting:

Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:hipmatch' and lookup table 'pan_vendor_info_lookup'.
Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:newapps' and lookup table 'pan_vendor_info_lookup'.
Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:system' and lookup table 'pan_vendor_info_lookup'.
Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:threat' and lookup table 'pan_vendor_info_lookup'.
Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:traffic' and lookup table 'pan_vendor_info_lookup'.
Info.csv being bloated by "lookup" log messages . Will not log additional errors. Refer search.log
The limit has been reached for log messages in info.csv. 131 messages have not been written to info.csv. Please refer to search.log for these messages or limits.conf to configure this limit.
The lookup table 'app_lookup' does not exist. It is referenced by configuration 'pan:threat'.
The lookup table 'app_lookup' does not exist. It is referenced by configuration 'pan:traffic'.
The lookup table 'classification_lookup' does not exist. It is referenced by configuration 'pan:hipmatch'.
The lookup table 'classification_lookup' does not exist. It is referenced by configuration 'pan:threat'.
The lookup table 'classification_lookup' does not exist. It is referenced by configuration 'pan:traffic'.
The lookup table 'endpoint_epm_lookup' does not exist. It is referenced by configuration 'pan:endpoint'.
The lookup table 'endpoint_log_subtypes_lookup' does not exist. It is referenced by configuration 'pan:endpoint'.
The lookup table 'endpoint_severity_lookup' does not exist. It is referenced by configuration 'pan:endpoint'.
The lookup table 'pan_vendor_action_lookup' does not exist. It is referenced by configuration 'pan:threat'.
The lookup table 'pan_vendor_action_lookup' does not exist. It is referenced by configuration 'pan:traffic'.
The lookup table 'sanctioned_saas_lookup' does not exist. It is referenced by configuration 'pan:threat'.
The lookup table 'sanctioned_saas_lookup' does not exist. It is referenced by configuration 'pan:traffic'.
The lookup table 'threat_lookup' does not exist. It is referenced by configuration 'pan:threat'.
[slsplunkind001p] Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:hipmatch' and lookup table 'pan_vendor_info_lookup'.
[slsplunkind001p] Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:newapps' and lookup table 'pan_vendor_info_lookup'.
0 Karma

dxmiller
Explorer

Below you'll see the output of the README.txt document located in $SPLUNK_HOME/etc/slave-apps/Splunk_TA_paloalto/ which shows that they're running the latest version.

[dmiller@slsplunkind001p etc]# cat /opt/splunk/etc/slave-apps/Splunk_TA_paloalto/README.txt
Palo Alto Networks Add-on for Splunk version 3.5.1

Copyright (C) 2009-2015 Splunk Inc. All Rights Reserved.

The Splunk Add-on for Palo Alto Networks allows a Splunk® Enterprise administrator to collect data from Palo Alto Networks Next-Generation Firewall devices and Advanced Endpoint Protection. The add-on collects traffic, threat, system, configuration, and endpoint logs from Palo Alto Networks physical or virtual firewall devices over syslog. After Splunk Enterprise indexes the events, you can consume the data using the pre-built dashboard panels included with the add-on. This add-on provides the inputs and CIM-compatible knowledge to use with other Splunk Enterprise apps, such as the Splunk App for Enterprise Security and the Splunk App for PCI Compliance.

Documentation for this add-on is located at: http://pansplunk.readthedocs.org/

For installation and set-up instructions, refer to the Getting Started section: http://pansplunk.readthedocs.org/en/latest/getting_started.html

For release notes, refer to the Release notes section: http://pansplunk.readthedocs.org/en/latest/release_notes.html[dmiller@slsplunkind001p etc]#

I'm not sure what else to do at this point or what other options we would have available.

0 Karma

kchamplin_splun
Splunk Employee
Splunk Employee

There are a few other items we can look at, the other things that come to mind are permissions on the lookups themselves both file system permissions and meta permissions. The default.meta that ships with the TA exports almost everything except the commands, it should look like:

#shared Application-level permissions
[]
access = read : [ * ], write : [ admin ]
export = system

#This is a TA, so export almost everything

[]
access = read : [ * ], write : [ admin, power ]
export = system

# Do not export commands
[commands]
export = none

Also does the process running Splunk have the proper permissions on the lookup files?

Lastly, can you look at the lookups settings in Splunk web - in a fresh install the sharing permissions are global.

0 Karma

kchamplin_splun
Splunk Employee
Splunk Employee

The only other thing I can think of is that the TA should be on the indexer if it is not already. The lookup "sanctioned_saas_lookup" is new to the latest app and TA, so it should definitely be finding that.

0 Karma