I'm following the instructions here and can't get it to even recognize the lookup. Did I miss something?
My transforms.conf:
[SUBJDECODE]
external_cmd = utfconv.py Subject
fields_list = Subject
My props.conf:
[source::/syslog/mail/*]
LOOKUP_table = SUBJDECODE Subject
Any search gives me the error: "The lookup table 'SUBJDECODE' does not exist. It is referenced by configuration 'source::/syslog/mail/*'."
I've even verified the lookup exists through the GUI -> Manager -> Lookups -> Lookup Defs
SUBJDECODE external No owner system Global | Permissions Enabled ....
It appears to recognize the props file, but is not fully integrating the transforms stanza. It shows in the GUI manager but can't be used. Both conf files are in $splunk/etc/system/local, but I've also tried them in the $splunk/etc/apps/search/local dir with equivalent results.
The stanza for the external lookup was not correct. The docs are ambiguous in a few places, and the absolutely terrible error message sent me on a wild goose chase, but I think I finally got there.
In transforms.conf you need to list the name of the field that will be handed to the lookup AS WELL AS the field name you want the script to output post-lookup. So:
[SUBJDECODE]
external_cmd = utfconv.py Subject decoded_subject
fields_list = Subject, decoded_subject
Even though decoded_subject doesn't exist, it needs to be there. I guess. Maybe. Anyway, it's working for me now. In my original stanza I was attempting to replace the original Subject field with the new value-- apparently a NOOP that blows up the logic and returns a completely unrelated error message.
To call the lookup, you need to leave off the output field (apparently):
source=*mail* | lookup SUBJDECODE Subject
Tada. It worked.
The stanza for the external lookup was not correct. The docs are ambiguous in a few places, and the absolutely terrible error message sent me on a wild goose chase, but I think I finally got there.
In transforms.conf you need to list the name of the field that will be handed to the lookup AS WELL AS the field name you want the script to output post-lookup. So:
[SUBJDECODE]
external_cmd = utfconv.py Subject decoded_subject
fields_list = Subject, decoded_subject
Even though decoded_subject doesn't exist, it needs to be there. I guess. Maybe. Anyway, it's working for me now. In my original stanza I was attempting to replace the original Subject field with the new value-- apparently a NOOP that blows up the logic and returns a completely unrelated error message.
To call the lookup, you need to leave off the output field (apparently):
source=*mail* | lookup SUBJDECODE Subject
Tada. It worked.
OK, I copied the dnslookup stanza from etc/system/default/transforms.conf and put it into local/transforms.conf. I named it dnslookup2. That works. So external lookups do work, but my custom command isn't working. That leads me to believe the error is with my script. If so, the error message provided is terribly misleading.
As for the script, running on the command line works fine. Piping CSV data into STDIN with the required args results in CSV being spit back out.
A few other things you may want to check here :
1) Where is the "utconfv.py" script located? As transforms.conf.spec states :
external_cmd = <string>
* Command and arguments to invoke to perform lookups.
* This string is parsed like a shell command.
* The first argument is expected to be a python script located in $SPLUNK_HOME/etc/<app_name>/bin (or ../etc/searchscripts) <=========
* Presence of this field indicates that lookup is external command based.
2) Are there no permission/ownership issues with utconf.py?
3) Check in $SPLUNK_HOME/var/log/splunk/python.log for errors referencing your lookup script.
The script is in $SPLUNK/etc/searchscripts and is set to 755. The python.log file is empty.
it might be an issue with your permissions? you can run:
splunk cmd btool transforms list --user=<user-running-search> --app=search --debug
and if it doesn't list the SUBJDECODE stanza, then it's a permissions issue w/ that particular user...
Done... yes the lookup stanza is there.
Now with shiny, new, strong, faster, better 4.1.5. Problem persists. 😞
Despite not being in the docs, I've added the metadata stanza (export=system). The stanza was already in the search app metadata. However, it was not in the system metadata file. I've added there also. Still no go. Anyone? Buehler?