Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to the search peer"

[jwightman2]

This warning is present in the Job control drop-down on search heads:
The 'minemeldfeeds_lookup' KV Store lookup table is empty or has not yet been replicated to the search peer (path used is: /opt/splunk/var/run/searchpeers/...)

The error is cited for 3 of the 5 indexers in a cluster, and the search heads are in a cluster.
Is there a resolution appropriate for this version or a step we missed?

Splunk v 7.2.5.1
Palo Alto Networks Add-on for Splunk v 6.1.1
Splunk Enterprise Security is in use, and there is no other Palo Alto "app" in place. (From the installation guide: "The Add-on can be used with or without the App.")
The add-on is in place on the search heads, indexers, and heavy forwarders.

One of the previous answers mentioned setting replicate=true ...
From what I can tell, that was already set by default in this version of the add-on due to these two excerpts:

Splunk_TA_paloalto/default/transforms.conf

[minemeldfeeds_lookup]
external_type = kvstore
collection = minemeldfeeds

Splunk_TA_paloalto/default/collections.conf

[minemeldfeeds]
replicate = true

Unlike other posts on answers.splunk, we did not have an upgrade involved. That made most of the recommendations and previously accepted answers unhelpful.

The kvstore migrate command did not seem to apply to this scenario and version; nothing I found suggested there was a way to force the knowledge bundle from search head to indexers (if that is the issue)- similar to the sync command available for kvstore, and shcluster-replicated-config.

Any tips would be appreciated.

[manikumarv]

Thanks to @fverdi for the tip.  The error seems to be due to the empty automatic lookup.   So I added a dummy entry into the minemeldfeeds kvstore collection to get rid of the warning message when searching the paloalto data in splunk.

Here's the curl command to add the entry.  Just remember to remove the entry if you enable the minemeld feeds. 

curl -k -u admin https://<SEARCH-HEAD>:8089/servicesNS/nobody/Splunk_TA_paloalto/storage/collections/data/minemeldfeeds -H "Content-Type: application/json" -d '{ "myKey": "temp" , "description": "remove this entry from this collection when enabling minemeld feeds"}

[jadamsplunk]

Hello,

I found this question while looking up a similar answer for my own custom KVStore. I don't have a solution for the Palo Alto addon directly, but I resolved my issues by adding append=True key_field=_key to my kvstore outputlookup.

What appears to be the issue in my case is that you need to specify key_field=<field> if you are are manually evaling your _key field and are outputting to an empty KVStore.

I would check to see if the KVStore output search his manually setting the key and not including that field. I've found if you remove the manual eval or add the key_field=<field>

You haven't had any answers or suggestions pertaining to your exact issue, but I hope this can help.

[splunk-the-prob]

Hi,

I recently came got annoyed of the problem and finally did some investigation.

Looks like the error is complaining about the kvstore of minemeldfeeds_lookup as being empty.

The original minemeldfeeds_lookup was trying to get results from sourcetype=pan:minemeld and we don't have a subscription for that.

Here's a workaround (referenced here @ https://docs.splunk.com/Documentation/Splunk/8.0.4/SearchReference/Outputlookup)

| makeresults | eval name="xyz" | eval token="12345"| outputlookup minemeldfeeds_lookup

This will add an entry and we won't have a blank file anymore.

Good luck and keep on splunking.

[jwightman2]

It looks like the same problem was reported on Github (for PaloAltoNetworks/SplunkforPaloAltoNetworks) on 2019/10/24. At the time of writing (2019/11/14), it also has no response.
https://github.com/PaloAltoNetworks/SplunkforPaloAltoNetworks/issues/95

[keith_d]

We're having the same error with 6.1.1 of the app and 7.3.3 of Splunk. I'd help if I could, but you're not alone.

[eugenek]

Have you found a solution?

[jwightman2]

From 2019/09/11, originally accidentally posted as an answer:
No solutions, good news, or follow-ups from any type of support personnel. (Modifications to post/formatting/title happened at the time of submission, possibly by evzhang_splunk.)

After letting a search job complete for a test today, the error is shown for all 5 indexers in the cluster.