topic Re: Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to in All Apps and Add-ons

Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to the search peer"

jwightman2 — Wed, 30 Sep 2020 00:46:24 GMT

This warning is present in the Job control drop-down on search heads:
The 'minemeldfeeds_lookup' KV Store lookup table is empty or has not yet been replicated to the search peer (path used is: /opt/splunk/var/run/searchpeers/...)

The error is cited for 3 of the 5 indexers in a cluster, and the search heads are in a cluster.
Is there a resolution appropriate for this version or a step we missed?

Splunk v 7.2.5.1
Palo Alto Networks Add-on for Splunk v 6.1.1
Splunk Enterprise Security is in use, and there is no other Palo Alto "app" in place. (From the installation guide: "The Add-on can be used with or without the App.")
The add-on is in place on the search heads, indexers, and heavy forwarders.

One of the previous answers mentioned setting replicate=true ...
From what I can tell, that was already set by default in this version of the add-on due to these two excerpts:

Splunk_TA_paloalto/default/transforms.conf

[minemeldfeeds_lookup]
external_type = kvstore
collection = minemeldfeeds

Splunk_TA_paloalto/default/collections.conf

[minemeldfeeds]
replicate = true

Unlike other posts on answers.splunk, we did not have an upgrade involved. That made most of the recommendations and previously accepted answers unhelpful.

The kvstore migrate command did not seem to apply to this scenario and version; nothing I found suggested there was a way to force the knowledge bundle from search head to indexers (if that is the issue)- similar to the sync command available for kvstore, and shcluster-replicated-config.

Any tips would be appreciated.

Re: Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to the search peer"

eugenek — Wed, 11 Sep 2019 16:59:27 GMT

Have you found a solution?

Re: Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to the search peer"

jwightman2 — Thu, 14 Nov 2019 18:59:05 GMT

From 2019/09/11, originally accidentally posted as an answer:
No solutions, good news, or follow-ups from any type of support personnel. (Modifications to post/formatting/title happened at the time of submission, possibly by evzhang_splunk.)

After letting a search job complete for a test today, the error is shown for all 5 indexers in the cluster.

Re: Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to the search peer"

jwightman2 — Thu, 14 Nov 2019 19:01:22 GMT

It looks like the same problem was reported on Github (for PaloAltoNetworks/SplunkforPaloAltoNetworks) on 2019/10/24. At the time of writing (2019/11/14), it also has no response.
https://github.com/PaloAltoNetworks/SplunkforPaloAltoNetworks/issues/95

Re: Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to the search peer"

keith_d — Mon, 13 Jan 2020 17:46:52 GMT

We're having the same error with 6.1.1 of the app and 7.3.3 of Splunk. I'd help if I could, but you're not alone.

Re: Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to the search peer"

jadamsplunk — Fri, 27 Mar 2020 13:07:03 GMT

Hello,

I found this question while looking up a similar answer for my own custom KVStore. I don't have a solution for the Palo Alto addon directly, but I resolved my issues by adding append=True key_field=_key to my kvstore outputlookup.

What appears to be the issue in my case is that you need to specify key_field=<field> if you are are manually evaling your _key field and are outputting to an empty KVStore.

I would check to see if the KVStore output search his manually setting the key and not including that field. I've found if you remove the manual eval or add the key_field=<field>

You haven't had any answers or suggestions pertaining to your exact issue, but I hope this can help.

Re: Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not ye

splunk-the-prob — Tue, 09 Jun 2020 21:23:33 GMT

Hi,

I recently came got annoyed of the problem and finally did some investigation.

Looks like the error is complaining about the kvstore of minemeldfeeds_lookup as being empty.

The original minemeldfeeds_lookup was trying to get results from sourcetype=pan:minemeld and we don't have a subscription for that.

Here's a workaround (referenced here @ https://docs.splunk.com/Documentation/Splunk/8.0.4/SearchReference/Outputlookup)

| makeresults | eval name="xyz" | eval token="12345"| outputlookup minemeldfeeds_lookup

This will add an entry and we won't have a blank file anymore.

Good luck and keep on splunking.

Re: Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to

manikumarv — Tue, 03 Nov 2020 17:12:40 GMT

Thanks to @fverdi for the tip. The error seems to be due to the empty automatic lookup. So I added a dummy entry into the minemeldfeeds kvstore collection to get rid of the warning message when searching the paloalto data in splunk.

Here's the curl command to add the entry. Just remember to remove the entry if you enable the minemeld feeds.

curl -k -u admin https://<SEARCH-HEAD>:8089/servicesNS/nobody/Splunk_TA_paloalto/storage/collections/data/minemeldfeeds -H "Content-Type: application/json" -d '{ "myKey": "temp" , "description": "remove this entry from this collection when enabling minemeld feeds"}