All Apps and Add-ons

Error at search time after upgrading Palo Alto Networks App for Splunk to version 6.0

veddersg
Explorer

After upgrading the Palo Alto Networks App for Splunk from version 5.4 to 6, I am getting the following error message from each member of the index cluster (3 member cluster).

[idx1] The lookup table 'minemeldfeeds_lookup' does not exist. It is referenced by configuration 'pan:threat'.
[idx1] The lookup table 'minemeldfeeds_lookup' does not exist. It is referenced by configuration 'pan:traffic'.

I have verified the latest SplunkforPaloAltoNetworks and Splunk_TA_paloalto are both installed on each indexer as well as the search head. How do I go about correcting this error?,I am getting the following errors from our Indexers (setup up in a 3 member cluster) after upgrading to version 6.0 from version 5.4.0.

[idx] The lookup table 'minemeldfeeds_lookup' does not exist. It is referenced by configuration 'pan:threat'.
[idx] The lookup table 'minemeldfeeds_lookup' does not exist. It is referenced by configuration 'pan:traffic'.

How do I correct these errors?

1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

You must enable replication for the kvstore collection [minemeldfeeds]

KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

Events of sourcetype pan:threat and pan:traffic have automatic lookups configured in props.conf / transforms.conf so the automatic lookup against the minemeldfeeds collection is running on the indexers but the minemeldfeeds kvstore collection doesn't exist there because it is not being replicated from the SH to the indexers by default.

ie:
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/props.conf

LOOKUP-minemeldfeeds_dest_lookup = minemeldfeeds_lookup indicator AS dest_ip OUTPUT value.autofocus_tags AS dest_autofocus_tags

/opt/splunk/etc/apps/Splunk_TA_paloalto/default/transforms.conf

'[minemeldfeeds_lookup]'
collection = minemeldfeeds
external_type = kvstore
fields_list = indicator, splunk_source, value.confidence, value.first_seen, value.last_seen, value.share_level, value.sources, value.type, value.autofocus_tags, value.autofocus_benign, value.autofocus_grayware, value.autofocus_malware, value.autofocus_num_matching_artifacts, value

/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf

[minemeldfeeds]
field.indicator = string
field.splunk_source = string
field.value = string

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

On your search heads create a file called collections.conf in $SPLUNK_HOME/etc/system/local/ (or within an app)
[minemeldfeeds]
replicate = true

Restart Splunk to apply your changes.
Once the next bundle is replicated to your indexers the kvstore collection definition will exist at the indexers and you should see this error message go away the next search you run.

If your indexers are running a version of Splunk Enterprise that is older than 6.3, attempts to run an automatic lookup fail with a "lookup does not exist" error. You must upgrade your indexers to 6.3 or later to use this functionality.

you can learn more about automatic lookups and kvstore collection replication here:
https://docs.splunk.com/Documentation/Splunk/7.0.0/Knowledge/Makeyourlookupautomatic

View solution in original post

hketer
Path Finder

Hi All,

I've ran a search and got these errors

hketer_0-1605204902945.png

 

I've checked the lookup and it's empty

hketer_1-1605204977481.png

Is there a way to recover it?

Thanks,

Hen

0 Karma

panguy
Contributor

Hi could you help me understand where these error messages are showing up? Are they showing up under 'Messages' from the navigation?

0 Karma

weicai88
Path Finder

rphillips:

I got the same error, and following your instructions, checked my settings with btool. Here's what's on my search heads:

/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf [minemeldfeeds]
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf enforceTypes = false
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf field.indicator = string
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf field.splunk_source = string
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf field.value = string
/opt/splunk/etc/system/default/collections.conf profilingEnabled = false
/opt/splunk/etc/system/default/collections.conf profilingThresholdMs = 1000
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf replicate = true
/opt/splunk/etc/system/default/collections.conf replication_dump_maximum_file_size = 10240
/opt/splunk/etc/system/default/collections.conf replication_dump_strategy = auto
/opt/splunk/etc/system/default/collections.conf type = undefined

As you can see, replicate = true. Why do i still see the error?

Thanks!
Wei

0 Karma

btorresgil
Builder

@weicai88: Have you installed the App and Add-on 6.0.1 or higher on all Searchheads, Indexers, and Heavy Forwarders?

0 Karma

weicai88
Path Finder

I only installed the Add-on 6.0.2, not the App. And yes, it is on both Search Heads and Indexers. It's odd that I have two Search Head clusters, but the errors only show up on one of them although the Add-on was installed on both.

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

The message will show up after you run a search if you haven't replicated the kvstore collection

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

You must enable replication for the kvstore collection [minemeldfeeds]

KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

Events of sourcetype pan:threat and pan:traffic have automatic lookups configured in props.conf / transforms.conf so the automatic lookup against the minemeldfeeds collection is running on the indexers but the minemeldfeeds kvstore collection doesn't exist there because it is not being replicated from the SH to the indexers by default.

ie:
/opt/splunk/etc/apps/Splunk_TA_paloalto/default/props.conf

LOOKUP-minemeldfeeds_dest_lookup = minemeldfeeds_lookup indicator AS dest_ip OUTPUT value.autofocus_tags AS dest_autofocus_tags

/opt/splunk/etc/apps/Splunk_TA_paloalto/default/transforms.conf

'[minemeldfeeds_lookup]'
collection = minemeldfeeds
external_type = kvstore
fields_list = indicator, splunk_source, value.confidence, value.first_seen, value.last_seen, value.share_level, value.sources, value.type, value.autofocus_tags, value.autofocus_benign, value.autofocus_grayware, value.autofocus_malware, value.autofocus_num_matching_artifacts, value

/opt/splunk/etc/apps/Splunk_TA_paloalto/default/collections.conf

[minemeldfeeds]
field.indicator = string
field.splunk_source = string
field.value = string

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

On your search heads create a file called collections.conf in $SPLUNK_HOME/etc/system/local/ (or within an app)
[minemeldfeeds]
replicate = true

Restart Splunk to apply your changes.
Once the next bundle is replicated to your indexers the kvstore collection definition will exist at the indexers and you should see this error message go away the next search you run.

If your indexers are running a version of Splunk Enterprise that is older than 6.3, attempts to run an automatic lookup fail with a "lookup does not exist" error. You must upgrade your indexers to 6.3 or later to use this functionality.

you can learn more about automatic lookups and kvstore collection replication here:
https://docs.splunk.com/Documentation/Splunk/7.0.0/Knowledge/Makeyourlookupautomatic

veddersg
Explorer

Thank you, that cleared it up for me.

0 Karma

rpquinlan
Path Finder

Thank you @rphillips - that was helpful. Quick clarification though..

You had said

On your search heads create a file called collections.conf in $SPLUNK_HOME/etc/system/local/ (or within an app)

Did you mean indexer (cluster)? Those error messages didn't go away for me until I did the above on the index cluster (That stanza in the collections.conf is also on my SHC).

Thanks!

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

@rpquinlan the configuration in collections.conf should be placed on the SHs. Upon the next knowledge bundle replication (SH-->Indexers) the kvstore collection will be replicated to the indexers. Its likely that you were still seeing the errors because a new knowledge bundle hadn't yet been sent to the indexers.

0 Karma

vgollapudi
Communicator

I was able to resolve this issue, here is the process I have followed

I have updated (replicate = true) parameter in collections.conf file on these instances (Cluster Search Heads Members as well as on Enterprise Search Head Instance).

Make sure you copy the collections.conf under local directory and make changes, /opt/splunk/etc/apps/Splunk_TA_paloalto/local/collections.conf

[minemeldfeeds]
replicate = true
field.indicator = string
field.splunk_source = string
field.value = string

Stanza should like above.

Push these changes on the Cluster Search Head through deployer such that you have similar changes across cluster.

Run the below command as per your environment
/opt/splunk/bin/splunk apply shcluster-bundle target {cluster search head captain} -auth

If the enterprise search head is managed by the deployment server then try to follow the below process

Edit the add-on collections.conf under local directory by making a local copy and reload the deployment server

/opt/splunk/bin/splunk reload deploy-server

This should fix the issue, you can also ensure whether the knowledge bundle is applied successfully or not in two ways.

One, try to search data through UI (Enterprise Search Head).

Second, login into your indexer server and cd this path /opt/splunk/etc/var/run/searchpeers/. Here you can see the Knowledge bundles that are applied and check if the latest bundle applied has the change that you have made which got either pushed through the deployment or deployer.

0 Karma

graju89
Path Finder

@vgollapudi Hi,

I am seeing same error after upgrading my splunk. Can you post your advise on this?
The error I am seeing on the serach head is
Could not load lookup=LOOKUP-minemeldfeeds_dest_lookup
Could not load lookup=LOOKUP-minemeldfeeds_src_lookup.
Search peer errors are same as above.

Note: search head is running 7.2 and search peers are running 6.6.6

0 Karma

btorresgil
Builder

Replication is on by default in the latest Add-on. Try re-installing the Add-on, it sounds like something is broken after the upgrade.

0 Karma

graju89
Path Finder

@btorresgil I tried re-installing too. Stil no luck

0 Karma

illao
New Member

@graju89 I've same issue just like yours and finally I found that the KV store is disabled on my search head. Once kv store enable and the Knowledge bundles is replicated to search peer, the error is gone.

0 Karma

vgollapudi
Communicator

Hello,

Did you follow the steps listed by me or rphilips? If you haven't do follow them, I did share the steps which helped me to resolve the error.

The Search Head version of our environment is 6.6.3. I'm unsure what has changed in 7.2 version.

0 Karma

graju89
Path Finder

@vgollapudi Yes, I followed it. But still I am seeing the error.

0 Karma

btorresgil
Builder

Did this suggestion fix the problem?

0 Karma

maciep
Champion

That means that you have an automatic lookup configured for those sourcetypes. But when the lookup referenced by the automatic lookup doesn't exist. You might want to start by seeing if that automatic lookup is actually defined in the latest version of the app or something that is left over.

But essentially, you either need to disable/remove the automatic lookup if it's not needed. OR find out why the lookup up it references is missing. For example, maybe a scheduled search creates that lookup and it hasn't run yet or hasn't produced any results

Settings -> Lookups -> Automatic Lookups (these are in props.conf)
This is where you define which fields to lookup, which to spit out and for which sourcetype

Settings -> Lookups -> Lookup definitions (these are in transforms.conf)
This is likely referenced by the automatic lookup and contains the lookup file info, and settings for lookup (like case sensitivity, wildcard matching, etc)

Settings -> Lookups -> Lookup table files (these are the actual files under /lookups)
This will show the path to the actual lookup file mentioned the lookup reference.

Side note: The reason the messages are coming from the indexers is not that the lookup is missing from the indexers app (under slaveapps) but that the search bundle pushed down to the indexers from the search head is missing the lookup. So one other possibility is that the lookup is blacklisted from the search bundle (distsearch.conf), but that's probably not too likely in this case.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...