Hi Dear Splunkers,
I have been working on creating a Custom TA for counting unicode characters for non-eng dataset (long story discussion post in PS), getting these lookup file errors
The Custom TA creation steps I followed: (on my personal laptop, installed bare-min fresh 9.3.2 enterprise trial version)
1) Created the custom TA named "TA-ucd" on app creation page (given read for all, execute for owner, shared with all apps)
2) created the ucd_category_lookup.py (made sure of the permissions)
$SPLUNK_HOME/etc/apps/TA-ucd/bin/ucd_category_lookup.py (this file should be readable and executable by the Splunk user, i.e. have at least mode 0500)
#!/usr/bin/env python import csv import unicodedata import sys def main(): if len(sys.argv) != 3: print("Usage: python category_lookup.py [char] [category]") sys.exit(1) charfield = sys.argv[1] categoryfield = sys.argv[2] infile = sys.stdin outfile = sys.stdout r = csv.DictReader(infile) header = r.fieldnames w = csv.DictWriter(outfile, fieldnames=r.fieldnames) w.writeheader() for result in r: if result[charfield]: result[categoryfield] = unicodedata.category(result[charfield]) w.writerow(result) main()
$SPLUNK_HOME/etc/apps/TA-ucd/default/transforms.conf
[ucd_category_lookup] external_cmd = ucd_category_lookup.py char category fields_list = char, category python.version = python3
$SPLUNK_HOME/etc/apps/TA-ucd/metadata/default.meta
[] access = read : [ * ], write : [ admin, power ] export = system
3) after creating the 3 files above mentioned, i did restart Splunk service and laptop as well.
4) still the search fails with lookup errors above mentioned.
5) source=*search.log* - does not produce anything (Surprisingly !)
Could you pls upvote the idea - https://ideas.splunk.com/ideas/EID-I-2176
PS - long story available here - https://community.splunk.com/t5/Splunk-Search/non-english-words-length-function-not-working-as-expec...
The lookup table 'ucd_count_chars_lookup' does not exist or is not available.
[...]
[ucd_category_lookup]
Are you sure those shouldn't match?
Thanks @PickleRick for your reply.
Hello Everyone.. i am pretty new for App dev, so pardon me, let me explain you step by step:
1) the external_cmd ---- ucd_category_lookup.py script is present...screenshot:
2) the lookup definition was not created previously, so just i created one. given permissions to search app, selected the ucd_category_lookup.py as the external command, added and verified the fields properly
3) lookup definition done, restart splunk done, but still same error.
4) i thought automatic lookups is what needed, created one:
5) restart Splunk done, still the same error.
OK. Several things here.
1. External lookup is not the same as external command. External lookup is a bit simpler version of an external command.
2. External lookup is _not_ the same as automatic lookup. An external lookup is a using SPL lookup command syntax to execute your external script while automatic lookup is a lookup which is automatically invoked on your data without the need of manually invoking the lookup command. The typical application of the automatic lookup is adjusting field values to CIM datamodel.
3. But still the lookup must match its definition so I pointed you to the fact that your stanza was named ucd_category_lookup but you were trying to use ucd_count_chars_lookup - these didn't match.
Did you read https://dev.splunk.com/enterprise/docs/devtools/externallookups/ ?
Thanks @PickleRick for the detailed ideas.
1) one issue - the transforms conf file was saved as transforms.conf.txt (thanks to windows ! ), corrected it.
2) now the search query is:
| makeresults
| eval _raw="இடும்பைக்கு"
| lookup ucd_category_lookup _raw output count
3) and the result is:
4) by that "Cannot find the source field '_raw' in the lookup table 'ucd_category_lookup',
i believe, the _raw field should be embedded inside the python script, but i am not sure.
Any help appreciated, karma points for sure. thanks.
The most important thing about writing an external lookup is here https://dev.splunk.com/enterprise/docs/devtools/externallookups/createexternallookup:
For each row in the input CSV table, populate the missing values. Then, to return this data to your search results, write each row in the output CSV table to the STDOUT output stream
In other words, the external lookup scripts gets CSV-formatted data on input, fills the gaps by whatever means necessary and returns the CSV on output from which splunkd performs "normal" lookup process.
So.
1. Just like with any lookup the fields you specify in fields_list in transforms.conf must match the fields you use in the lookup command in SPL. If they don't, you have to use the AS clause.
2. The fields in fields_list must be properly processed and returned by the lookup script. The explicit names of the fields in case of the example external lookup is in fact not strictly necessary
external_cmd = external_lookup.py clienthost clientip
In this case the "clienthost clientip" part is just a list of parameters accepted by the external_lookup.py script because someone wrote the script itself as accepting dynamically specified column names. If those were hardcoded at the script level (always processing the "clienthist" and "clientip" columns from the input CSV stream) you could define it simply as
external_cmd = external_lookup.py
So the minimal version of the working external lookup for returning the length of the field called "data" should look (with one caveat explained later) like this:
transforms.conf:
[test_lenlookup]
external_cmd = lenlookup.py
fields_list = data, length
python.version = python3
And the lenlookup.py file itself:
#!/usr/bin/env python3
import csv
import sys
def main():
infile = sys.stdin
outfile = sys.stdout
r = csv.DictReader(infile)
w = csv.DictWriter(outfile, fieldnames=["data","length"])
w.writeheader()
for result in r:
if result["data"]:
result["length"]=len(result["data"])
w.writerow(result)
main()
Yes, it doesn't do any sanity checking or error handling but it does work for something like
| makeresults
| eval data="whatever"
| lookup test_lenlookup data
Of course this is a simple len() python function which in your case might or might not be what you need so the core functionality you might need to rewrite on your own.
One important caveat. Even thought the spec file for transforms .conf says
external_cmd = <string> * Provides the command and arguments to invoke to perform a lookup. Use this for external (or "scripted") lookups, where you interface with with an external script rather than a lookup table. * This string is parsed like a shell command. * The first argument is expected to be a python script (or executable file) located in $SPLUNK_HOME/etc/apps/<app_name>/bin. * Presence of this field indicates that the lookup is external and command based. * Default: empty string
I was unable to run my external lookup when the script was placed anywhere else than $SPLUNK_HOME/etc/system/bin. Judging from Answers history it seems to be some kind of a bug.
EDIT: OK. I found it. It seems that for an external lookup to work you must give permissions to both the lookup definition (which you may as well do in WebUI) as well as to the script file itself (which you must do using the .meta file.
So in this case you need something like this:
[bin/lenlookup.py]
access= read : [*]
export = system
[transforms/test_lenlookup]
access = read : [*]
export = system
Thanks for your detailed reply. Created the python script, added the transforms.conf file, added the lookup definition, updated the meta file, and restarted Splunk and then ran:
| makeresults
| eval data="இடும்பைக்கு"
| lookup test_lenlookup data
Still the error:
looks like the issue is solved 90%, but some basic issues here, but not sure of what. ok, let me revisit the whole issue tomorrow, thanks a lot for your help.
Check permissions. Both in Splunk (the meta entries) an on filesystem.
Check if your script runs if you call it with
splunk cmd <app_path>/bin/lenlookup.py
Ok, does your lookup definition list _raw as one of input fields? If not you must use the AS clause for input field.
able to get rid of one error: