topic Re: External lookups: lookup not found error in Splunk Search

External lookups: lookup not found error

twinspop — Fri, 17 Sep 2010 04:03:14 GMT

I'm following the instructions here and can't get it to even recognize the lookup. Did I miss something?

My transforms.conf:

[SUBJDECODE]
external_cmd = utfconv.py Subject
fields_list = Subject

My props.conf:

[source::/syslog/mail/*]
LOOKUP_table = SUBJDECODE Subject

Any search gives me the error: "The lookup table 'SUBJDECODE' does not exist. It is referenced by configuration 'source::/syslog/mail/*'."

I've even verified the lookup exists through the GUI -> Manager -> Lookups -> Lookup Defs

SUBJDECODE   external   No owner   system   Global | Permissions   Enabled ....

It appears to recognize the props file, but is not fully integrating the transforms stanza. It shows in the GUI manager but can't be used. Both conf files are in $splunk/etc/system/local, but I've also tried them in the $splunk/etc/apps/search/local dir with equivalent results.

Re: External lookups: lookup not found error

twinspop — Sat, 18 Sep 2010 04:39:56 GMT

Despite not being in the docs, I've added the metadata stanza (export=system). The stanza was already in the search app metadata. However, it was not in the system metadata file. I've added there also. Still no go. Anyone? Buehler?

Re: External lookups: lookup not found error

twinspop — Sat, 18 Sep 2010 05:21:05 GMT

Now with shiny, new, strong, faster, better 4.1.5. Problem persists. 😞

Re: External lookups: lookup not found error

sophy — Sat, 18 Sep 2010 06:39:22 GMT

it might be an issue with your permissions? you can run:

splunk cmd btool transforms list --user=<user-running-search> --app=search --debug

and if it doesn't list the SUBJDECODE stanza, then it's a permissions issue w/ that particular user...

Re: External lookups: lookup not found error

twinspop — Tue, 21 Sep 2010 04:45:06 GMT

Done... yes the lookup stanza is there.

Re: External lookups: lookup not found error

hexx — Tue, 21 Sep 2010 07:23:13 GMT

A few other things you may want to check here :

1) Where is the "utconfv.py" script located? As transforms.conf.spec states :

external_cmd = <string>
* Command and arguments to invoke to perform lookups.
* This string is parsed like a shell command.
* The first argument is expected to be a python script located in $SPLUNK_HOME/etc/<app_name>/bin (or ../etc/searchscripts) <=========
* Presence of this field indicates that lookup is external command based.

2) Are there no permission/ownership issues with utconf.py?

3) Check in $SPLUNK_HOME/var/log/splunk/python.log for errors referencing your lookup script.

Re: External lookups: lookup not found error

twinspop — Tue, 21 Sep 2010 21:24:53 GMT

The script is in $SPLUNK/etc/searchscripts and is set to 755. The python.log file is empty.

Re: External lookups: lookup not found error

twinspop — Tue, 21 Sep 2010 21:55:49 GMT

OK, I copied the dnslookup stanza from etc/system/default/transforms.conf and put it into local/transforms.conf. I named it dnslookup2. That works. So external lookups do work, but my custom command isn't working. That leads me to believe the error is with my script. If so, the error message provided is terribly misleading.

As for the script, running on the command line works fine. Piping CSV data into STDIN with the required args results in CSV being spit back out.

Re: External lookups: lookup not found error

twinspop — Tue, 21 Sep 2010 23:06:47 GMT

The stanza for the external lookup was not correct. The docs are ambiguous in a few places, and the absolutely terrible error message sent me on a wild goose chase, but I think I finally got there.

In transforms.conf you need to list the name of the field that will be handed to the lookup AS WELL AS the field name you want the script to output post-lookup. So:

[SUBJDECODE]
external_cmd = utfconv.py Subject decoded_subject
fields_list = Subject, decoded_subject

Even though decoded_subject doesn't exist, it needs to be there. I guess. Maybe. Anyway, it's working for me now. In my original stanza I was attempting to replace the original Subject field with the new value-- apparently a NOOP that blows up the logic and returns a completely unrelated error message.

To call the lookup, you need to leave off the output field (apparently):

source=*mail* | lookup SUBJDECODE Subject

Tada. It worked.