topic Re: Why don't my custom lookups work after upgrading to 4.1? in Splunk Search

Why don't my custom lookups work after upgrading to 4.1?

Lowell — Wed, 07 Apr 2010 05:00:09 GMT

I recently upgrade a test system to Splunk 4.1, and my lookups are all giving me the following error:

The lookup table 'MyFieldLookup' does not exist. It is referenced by configuration 'my_source_type'.

I get this error appears is a flashy read box on all searches in any app except for the app where MyFieldLookup is defined. (MyFieldLookup is exported globally.) This occurs even with searches that are not using any fields related to the defined lookup, even if when searching on sourcetypes that are not associated with the lookup.

Are there a known issues with Lookups beyond what's listed Known issues page? (I tried changing the EOLs from CR/LF to LF, but that had no effect, based on the listed SPL-29434)

I have a single lookup defined which was working fine on Splunk 4.0.10.

I suspect that there could be some kind of permissions issue here, since the lookup works when I'm within the context of that app, but as soon as I search from a different app (which is the normal case based on my setup) then I get the error shown above. I've double checked all my permissions, and even added an explicit entry for [lookups/my_lookup_file.csv], but nothing seems to make a difference.

If I run: btool transforms list --app=<app-1> MyFieldLookup, (where app-1 is where the config is defined) I see the configuration entry, but if I run the command with --app=<app-2> (the app that most frequently uses the lookup), then there is no output, so it would appear that splunk cannot see that entry (or any other entry in the transforms.conf file)

My default.meta includes:

[transforms]
access = read : [ * ], write : [ admin ]
owner = nobody
export = system

So the transforms entries should be visible from other apps.

Update: I found out that btool seems to behave differently if you specify a user. For example, if I run btool transforms list --app=<app-1> --user=someuser MyFieldLookup, now I get the results I was expecting. So my problem wasn't with my transforms.conf entry or it's permissions.

Re: Why don't my custom lookups work after upgrading to 4.1?

gkanapathy — Wed, 07 Apr 2010 05:10:57 GMT

Lookups and searchscripts are now permissioned in the metadata/*.meta file(s). It should work with a stanza like:

[lookups]
export=system

[lookups/MyFieldLookup]
export=system

You should use the transforms.conf-defined name of the lookup to qualify it, not the file name.

Incidentally, I would use only local.meta for my own apps, not default.meta, or you'll wind up having to manage two files if you edit permissions through the GUI.

Re: Why don't my custom lookups work after upgrading to 4.1?

Lowell — Wed, 07 Apr 2010 05:42:32 GMT

I think you are mixing two things here. I think when you said "[lookups/MyFieldLookup]", you mean "[transforms/MyFieldLookup]". The [lookups/...] entries seem to be for specific files in the lookups folder, not the name of the lookup (which is in the transforms file). I think my actual problem was I was some how missing the export=system in my [lookups] entry.

Re: Why don't my custom lookups work after upgrading to 4.1?

William — Wed, 07 Apr 2010 17:09:29 GMT

I met this problem too.

The lookup works in the flashtimeline view of my app but does NOT work in any other of my savedsearch/dashboard/view of that app. The error message is the same: The lookup table xxxxx does not exist.

By referring the question and answer, I've solved my problem by using "export = system" of [lookups/...] and [transforms/...]. Their original values are "export = none".

I am still confusing why it should export the objects (lookup file and transform) to "system" for a lookup which is only supposed to be used in its app?

Re: Why don't my custom lookups work after upgrading to 4.1?

Lowell — Thu, 08 Apr 2010 00:21:55 GMT

You are correct, you shouldn't have to export your objects to use them within that app. In my original scenario, my lookups worked within the app they were defined in, but not from another app. You may want to check your settings again, it's really easy to overlook some of the permissions settings.

Re: Why don't my custom lookups work after upgrading to 4.1?

Lowell — Thu, 08 Apr 2010 01:42:49 GMT

Since I posted my last comment. I too am running to an issue where I can't access an object shared at the application level. The only way to access it seems to be to set the access-level to global... very weird. (I'm seeing the same behavior in Splunk 4.0, BTW)

Re: Why don't my custom lookups work after upgrading to 4.1?

Lowell — Tue, 13 Apr 2010 02:51:27 GMT

For the simplicity, let me consolidate the answers and comments here:

NOTICE: Splunk 4.1.1 and later add a migration note about these permissions changes when during the upgrade. So hopefully fewer users will run into this issue. The migration notes point your to the following docs page.

In Splunk 4.1. it is now possible to set permissions on individual lookup tables (aka .csv files) in the lookups folder of an app. Because of this new control in Splunk 4.1, you have the option to grant or restrict access to these files. There are actually 3 different sets of permissions to control access to the lookups now.

You basically have two options: (1) Disable your lookups to make the error message go away, and use the lookup search operator instead. Or (2), make your transforms and lookups global (or otherwise tweak your permissions till you get it all working again...)

Option 1: Disabling your automatic lookup

Assuming your have sensitive info in your lookup file, you can avoid this error message by disabling (comment-out) or changing the permissions of your LOOKUP-<class> entry in your props.conf file. You can do this from the UI via Manager » Lookups » Automatic lookups, then find your stanza name (which will be based on your source, sourcetype, or host and the name of your LOOKUP line) and change the permission using the Permissions link.

If you prefer editing config files directly, you can also set your permissions in metadata/local.meta with an entry like this:

[props/my_sourcetype/LOOKUP-MyFieldLookup]
access = read : [ *  ], write : [ admin ]
export = none

Keep in mind, that if you choose to completely remove (or comment out) your LOOKUP entries, rather than change the permissions, then you will need to use the lookup search command now, since the lookup fields will no longer be "automatic".

Option 2: Enable global lookup access

You can go back to the globally-available behavior of Splunk 4.0 by setting your lookups permissions using the UI (Manger >> Lookups >> Lookup Table files, then click on the Permissions link).

This can also be done by updating the metadata/local.meta file manually. To allow all access to all csv files from any application, add an entry like so:

[lookups]
export = system

Or, for a single lookup file:

[lookups/my_lookup_file.csv]
export = system

NOTE: You can also specify permission to the Lookup definition (based on the transforms.conf name), in same way as in Splunk 4.0. So if you want your lookup to be global, you would use something like:

[tranforms/MyFieldLookup]
export=system

Again, this can also be done from the UI. In Splunk 4.1, navigate to Manger >> Lookups >> Lookup definitions, then click the Permissions link.

Re: Why don't my custom lookups work after upgrading to 4.1?

ndoshi — Wed, 14 Apr 2010 06:57:40 GMT

I don't think the export is working properly globally. To use your example, if I do

[lookups/my_lookup_file.csv]
export = system

in my local.meta file for the app, the error message disappears as noted. However, as soon I move to another app that doesn't even use the my_lookup_file.csv and its associated props.conf and transforms.conf files, the error:

The lookup table 'MyFieldLookup' does not exist. It is referenced by configuration 'my_source_type'.

returns when loading a dashboard from within another app. The workaround is to comment out the lookup entry in props.conf to disable automatic lookups for this lookup. You could still call the lookup explicitly within a search using the lookup command. I have not made my transforms global within local.meta, which may solve this, because I do not want to make these entries global.

Re: Why don't my custom lookups work after upgrading to 4.1?

Lowell — Wed, 14 Apr 2010 07:57:15 GMT

Yes, thats the same issue I was seeing (I updated the question to make the point about any search no matter how unrelated results in this error message.) As you point out, you certainly do have two choices. Either make it all global, or hide the transformer section and remove the lookups altogether and do it via the search lookup command.

Re: Why don't my custom lookups work after upgrading to 4.1?

Lowell — Wed, 14 Apr 2010 08:49:38 GMT

I updated my consolidated answer to include your input. Thanks.

Re: Why don't my custom lookups work after upgrading to 4.1?

ndoshi — Fri, 16 Apr 2010 01:12:35 GMT

I just wanted to add one more piece of detail to Lowell's option 2. If you simply do:

[lookups/my_lookup_file.csv]
export = system

within an app's local.meta file for automatic lookup, this will let you configure props.conf lookup within the app without issues. This assumes that you will only use this app and not use any other app.

As soon as you move to another app (e.g., start using the search app and then use the Windows app), you will get the error below within the other app's dashboards.

The lookup table 'MyFieldLookup' does not exist. It is referenced by configuration 'my_source_type'.

If you want to avoid that, you have to enable your corresponding transforms to be exported to global. In the context of this example, the original app's local.meta file should be:

[lookups/my_lookup_file.csv]
export = system

[tranforms/MyFieldLookup]
export = system

After doing this, you can use any other app within Splunk and not receive the error.

Re: Why don't my custom lookups work after upgrading to 4.1?

dwaddle — Sat, 28 Jun 2014 22:54:23 GMT

With 6.1.1, for a dynamic lookup you have to export both [transforms] and [searchscripts] to system in order to make your dynamic lookup usable across apps