Splunk Search

[indexer1] The lookup table 'XXX' does not exist. It is referenced by configuration 'YYY'.

stepheneardley
Explorer

I've been investigating this error which is appearing 6 times per search (1 for each indexer in the cluster) for a while now and have come to an end pass... This is happening for all non Admin roles.

The permissions are global (all apps) for both the lookup file and the definition with everyone having read access to both. This is all configured on the search head cluster.

The actual lookup csv file exists on all search heads in the search head cluster and are owned by splunk:splunk.

I read on a few other questions that the lookup definition should also be deployed to the indexer cluster but I'm not sure why this would matter. In my case this error is appearing for searches that aren't even referencing the lookup in question. Do I really need to configure and schedule the search which creates the lookup for the search head cluster on the indexer cluster as well?

search.log on the search head

09-11-2017 13:15:42.187 WARN LookupOperator - Unable to find property=filename for lookup=XXX will attempt to use implicit filename.
09-11-2017 13:15:42.187 WARN LookupOperator - No valid lookup found for lookup=XXX
09-11-2017 13:15:42.187 ERROR LookupOperator - The lookup table 'XXX' does not exist. It is referenced by configuration 'YYY'.
09-11-2017 13:15:42.187 ERROR LookupOperator - The lookup table 'XXX' does not exist. It is referenced by configuration 'YYY'.

search.log from an indexer

09-11-2017 13:15:42.338 WARN LookupOperator - Unable to find property=filename for lookup=XXX will attempt to use implicit filename.
09-11-2017 13:15:42.338 WARN LookupOperator - No valid lookup found for lookup=XXX
09-11-2017 13:15:42.338 ERROR LookupOperator - The lookup table 'XXX' does not exist. It is referenced by configuration 'YYY'.
09-11-2017 13:15:42.338 ERROR LookupOperator - The lookup table 'XXX' does not exist. It is referenced by configuration 'YYY'.

Has anyone got any ideas on what could be causing this or where to look next?

0 Karma

woodcock
Esteemed Legend

You have an automatic lookup named YYY that is referencing a lookup definition named XXX that either does not exist at all or exists at a privilege level that is lower than that of the automatic lookup definition. Try this search to find the details of the detached automatic lookup:

|rest/servicesNS/-/-/data/props/lookups splunk_server=local
| search title="YYY"
| dedup id
| table eai:acl.app eai:acl.owner eai:acl.sharing title overwrite value *
| rename eai:acl.* AS *
| sort 0 title

Use this search to find the details of the detached lookup definition (if it exists):

|rest/servicesNS/-/-/data/transforms/lookups splunk_server=local
| search title="XXX"
| dedup id
| table eai:acl.app eai:acl.owner eai:acl.sharing disabled title type filename match_type case_sensitive_match fields_list *
| rename eai:acl.* AS *
| sort 0 title
0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

@stepheneardley Its likely that you have an automatic lookup configured on the indexers and the .csv is being blacklisted from the SH bundle which is distributed to the indexers. A few things you can look at:

1] what files are being blacklisted from the knowledge bundle sent from SH--> indexer
on the search head:
$SPLUNK_HOME/bin
./splunk btool distsearch list replicationBlacklist --debug

To fix the issue you'll want to remove the blacklist for the lookup in question

2] check what automatic lookups are configured on your indexer

| rest splunk_server=IDX1 /servicesNS/-/-/configs/conf-props/ | fieldsummary | search field=LOOKUP-* | rex field=field "LOOKUP-(?<LOOKUP_name_props>\S+)" | rex field=values "\[\{\"value\"\:\"(?<lookup_value_props>.+)\"\," | rex field=lookup_value_props "(?<transforms_stanza>\S+)" | join transforms_stanza [| rest splunk_server=IDX1 /servicesNS/-/-/configs/conf-transforms/ | search filename=*.csv | rename eai:appName as app | rex field=id "(?<transforms_stanza>(?:[^\/\r\n]*))$" | eval stanza=urldecode(transforms_stanza)] | table app transforms_stanza filename LOOKUP_name_props lookup_value_props

note: for splunk_server replace IDX1 with your indexer name

_smp_
Builder

This was the right answer for me. I had blacklisted the apps/*/bin/... directory due to some very large CSV lookup files. However, the Splunk_TA_windows addon includes a scripted input in the bin directory that also getting blacklisted. This caused my indexers to generate an error about a missing lookup in that app. I had to modify my custom blacklist regex to exempt the apps/Splunk_TA_window/bins directory from the blacklist. That got rid of the error for me.

0 Karma

stepheneardley
Explorer

In the results of the btool I spotted this. It's difficult to read but it looks like that should only exclude *.tmp and *.index files from /lookups/ in system/, app/ and users/ from the SH bundles. Right?

lookupindexfiles = (system|apps/|users(/_reserved)?//)/lookups/.(tmp$|index($|/...))

I tried commenting out this line and restarted the search head cluster. I waited around 30 minutes just to allow for replication and what not but the errors persist.

Also wouldn't this affect all users of the system and not just admin users?

Our lookups live in $SPLUNK_HOME/etc/apps/OurApp/lookups/XXX.csv on the search heads so they should be included in the bundles. I can see references to the lookup file when I search the indexers;

splunk@Indexer1:~> find ./ -iname xxx.csv
./var/run/searchpeers/34C862E5-CF2F-4222-A10C-C675B63D2D2D-1505214916/apps/OurApp/lookups/xxx.csv
./var/run/searchpeers/34C862E5-CF2F-4222-A10C-C675B63D2D2D-1505181906/apps/OurApp/lookups/xxx.csv
./var/run/searchpeers/34C862E5-CF2F-4222-A10C-C675B63D2D2D-1505174473/apps/OurApp/lookups/xxx.csv
./var/run/searchpeers/34C862E5-CF2F-4222-A10C-C675B63D2D2D-1505196072/apps/OurApp/lookups/xxx.csv
./var/run/searchpeers/34C862E5-CF2F-4222-A10C-C675B63D2D2D-1505185270/apps/OurApp/lookups/xxx.csv

I tried running the rest command you provided which returned 0 results. I stripped it back to "| search field=LOOKUP-*" which only returned 1 result - LOOKUP-dmc_add_instance_info.

I really appreciate your help with this. Any other thoughts?

0 Karma

anshul0915
Explorer

Did you the fix for it . I am using splunk version 6.5.2 facing the same issue for no admin role the error is coming in dashboard that lookup does not exists , i check all the permission for lookup file , lookup definition and automatic look each having read permission for every one

0 Karma

stepheneardley
Explorer

I haven't figured out why it is behaving this way yet but when I changed the lookup's permissions to the scope of the app that it was created in the error disappeared for other users. I still don't understand why Splunk doesn't like it being shared globally. It may be related to metadata permissions however I haven't had time to look into it properly. We use strict access controls for our apps using the metadata files.

0 Karma

anshul0915
Explorer

I also get figured it out :
When i changed the app of look up file to dame as look up defination and automatic look up the issue disappeared .The user with non admin role is dont have access to app in which lookup file was created so i make this to in search

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...