Splunk Search
Highlighted

External lookups: lookup not found error

Influencer

I'm following the instructions here and can't get it to even recognize the lookup. Did I miss something?

My transforms.conf:

[SUBJDECODE]
external_cmd = utfconv.py Subject
fields_list = Subject

My props.conf:

[source::/syslog/mail/*]
LOOKUP_table = SUBJDECODE Subject

Any search gives me the error: "The lookup table 'SUBJDECODE' does not exist. It is referenced by configuration 'source::/syslog/mail/*'."

I've even verified the lookup exists through the GUI -> Manager -> Lookups -> Lookup Defs

SUBJDECODE   external   No owner   system   Global | Permissions   Enabled ....

It appears to recognize the props file, but is not fully integrating the transforms stanza. It shows in the GUI manager but can't be used. Both conf files are in $splunk/etc/system/local, but I've also tried them in the $splunk/etc/apps/search/local dir with equivalent results.

Tags (1)
Highlighted

Re: External lookups: lookup not found error

Influencer

Despite not being in the docs, I've added the metadata stanza (export=system). The stanza was already in the search app metadata. However, it was not in the system metadata file. I've added there also. Still no go. Anyone? Buehler?

0 Karma
Highlighted

Re: External lookups: lookup not found error

Influencer

Now with shiny, new, strong, faster, better 4.1.5. Problem persists. 😞

0 Karma
Highlighted

Re: External lookups: lookup not found error

Splunk Employee
Splunk Employee

it might be an issue with your permissions? you can run:

splunk cmd btool transforms list --user=<user-running-search> --app=search --debug

and if it doesn't list the SUBJDECODE stanza, then it's a permissions issue w/ that particular user...

Highlighted

Re: External lookups: lookup not found error

Influencer

Done... yes the lookup stanza is there.

0 Karma
Highlighted

Re: External lookups: lookup not found error

Splunk Employee
Splunk Employee

A few other things you may want to check here :

1) Where is the "utconfv.py" script located? As transforms.conf.spec states :

external_cmd = <string>
* Command and arguments to invoke to perform lookups.
* This string is parsed like a shell command.
* The first argument is expected to be a python script located in $SPLUNK_HOME/etc/<app_name>/bin (or ../etc/searchscripts) <=========
* Presence of this field indicates that lookup is external command based.

2) Are there no permission/ownership issues with utconf.py?

3) Check in $SPLUNK_HOME/var/log/splunk/python.log for errors referencing your lookup script.

Highlighted

Re: External lookups: lookup not found error

Influencer

The script is in $SPLUNK/etc/searchscripts and is set to 755. The python.log file is empty.

0 Karma
Highlighted

Re: External lookups: lookup not found error

Influencer

OK, I copied the dnslookup stanza from etc/system/default/transforms.conf and put it into local/transforms.conf. I named it dnslookup2. That works. So external lookups do work, but my custom command isn't working. That leads me to believe the error is with my script. If so, the error message provided is terribly misleading.

As for the script, running on the command line works fine. Piping CSV data into STDIN with the required args results in CSV being spit back out.

0 Karma
Highlighted

Re: External lookups: lookup not found error

Influencer

The stanza for the external lookup was not correct. The docs are ambiguous in a few places, and the absolutely terrible error message sent me on a wild goose chase, but I think I finally got there.

In transforms.conf you need to list the name of the field that will be handed to the lookup AS WELL AS the field name you want the script to output post-lookup. So:

[SUBJDECODE]
external_cmd = utfconv.py Subject decoded_subject
fields_list = Subject, decoded_subject

Even though decoded_subject doesn't exist, it needs to be there. I guess. Maybe. Anyway, it's working for me now. In my original stanza I was attempting to replace the original Subject field with the new value-- apparently a NOOP that blows up the logic and returns a completely unrelated error message.

To call the lookup, you need to leave off the output field (apparently):

source=*mail* | lookup SUBJDECODE Subject

Tada. It worked.

View solution in original post