All Apps and Add-ons

Palo Alto Networks for Splunk v 6.1.1: Receiving error "lookup table is empty or has not yet been replicated to the search peer"

jwightman2
Explorer

This warning is present in the Job control drop-down on search heads:
The 'minemeldfeeds_lookup' KV Store lookup table is empty or has not yet been replicated to the search peer (path used is: /opt/splunk/var/run/searchpeers/...)

The error is cited for 3 of the 5 indexers in a cluster, and the search heads are in a cluster.
Is there a resolution appropriate for this version or a step we missed?

Splunk v 7.2.5.1
Palo Alto Networks Add-on for Splunk v 6.1.1
Splunk Enterprise Security is in use, and there is no other Palo Alto "app" in place. (From the installation guide: "The Add-on can be used with or without the App.")
The add-on is in place on the search heads, indexers, and heavy forwarders.

One of the previous answers mentioned setting replicate=true ...
From what I can tell, that was already set by default in this version of the add-on due to these two excerpts:

Splunk_TA_paloalto/default/transforms.conf

[minemeldfeeds_lookup]
external_type = kvstore
collection = minemeldfeeds

Splunk_TA_paloalto/default/collections.conf

[minemeldfeeds]
replicate = true

Unlike other posts on answers.splunk, we did not have an upgrade involved. That made most of the recommendations and previously accepted answers unhelpful.

The kvstore migrate command did not seem to apply to this scenario and version; nothing I found suggested there was a way to force the knowledge bundle from search head to indexers (if that is the issue)- similar to the sync command available for kvstore, and shcluster-replicated-config.

Any tips would be appreciated.

manikumarv
Explorer

Thanks to @fverdi for the tip.  The error seems to be due to the empty automatic lookup.   So I added a dummy entry into the minemeldfeeds kvstore collection to get rid of the warning message when searching the paloalto data in splunk.

Here's the curl command to add the entry.  Just remember to remove the entry if you enable the minemeld feeds. 

curl -k -u admin https://<SEARCH-HEAD>:8089/servicesNS/nobody/Splunk_TA_paloalto/storage/collections/data/minemeldfeeds -H "Content-Type: application/json" -d '{ "myKey": "temp" , "description": "remove this entry from this collection when enabling minemeld feeds"}

jadamsplunk
Path Finder

Hello,

I found this question while looking up a similar answer for my own custom KVStore. I don't have a solution for the Palo Alto addon directly, but I resolved my issues by adding append=True key_field=_key to my kvstore outputlookup.

What appears to be the issue in my case is that you need to specify key_field=<field> if you are are manually evaling your _key field and are outputting to an empty KVStore.

I would check to see if the KVStore output search his manually setting the key and not including that field. I've found if you remove the manual eval or add the key_field=<field>

You haven't had any answers or suggestions pertaining to your exact issue, but I hope this can help.

splunk-the-prob
Engager

Hi,

I recently came got annoyed of the problem and finally did some investigation.

Looks like the error is complaining about the kvstore of minemeldfeeds_lookup as being empty.

 

The original minemeldfeeds_lookup was trying to get results from sourcetype=pan:minemeld and we don't have a subscription for that.

Here's a workaround (referenced here @ https://docs.splunk.com/Documentation/Splunk/8.0.4/SearchReference/Outputlookup)

| makeresults | eval name="xyz" | eval token="12345"| outputlookup minemeldfeeds_lookup

This will add an entry and we won't have a blank file anymore.

 

Good luck and keep on splunking.

 

jwightman2
Explorer

It looks like the same problem was reported on Github (for PaloAltoNetworks/SplunkforPaloAltoNetworks) on 2019/10/24. At the time of writing (2019/11/14), it also has no response.
https://github.com/PaloAltoNetworks/SplunkforPaloAltoNetworks/issues/95

0 Karma

keith_d
Explorer

We're having the same error with 6.1.1 of the app and 7.3.3 of Splunk. I'd help if I could, but you're not alone.

0 Karma

eugenek
Path Finder

Have you found a solution?

0 Karma

jwightman2
Explorer

From 2019/09/11, originally accidentally posted as an answer:
No solutions, good news, or follow-ups from any type of support personnel. (Modifications to post/formatting/title happened at the time of submission, possibly by evzhang_splunk.)

After letting a search job complete for a test today, the error is shown for all 5 indexers in the cluster.

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...