The most important thing about writing an external lookup is here https://dev.splunk.com/enterprise/docs/devtools/externallookups/createexternallookup: For each row in the input CSV table, populate...
See more...
The most important thing about writing an external lookup is here https://dev.splunk.com/enterprise/docs/devtools/externallookups/createexternallookup: For each row in the input CSV table, populate the missing values. Then, to return this data to your search results, write each row in the output CSV table to the STDOUT output stream In other words, the external lookup scripts gets CSV-formatted data on input, fills the gaps by whatever means necessary and returns the CSV on output from which splunkd performs "normal" lookup process. So. 1. Just like with any lookup the fields you specify in fields_list in transforms.conf must match the fields you use in the lookup command in SPL. If they don't, you have to use the AS clause. 2. The fields in fields_list must be properly processed and returned by the lookup script. The explicit names of the fields in case of the example external lookup is in fact not strictly necessary external_cmd = external_lookup.py clienthost clientip In this case the "clienthost clientip" part is just a list of parameters accepted by the external_lookup.py script because someone wrote the script itself as accepting dynamically specified column names. If those were hardcoded at the script level (always processing the "clienthist" and "clientip" columns from the input CSV stream) you could define it simply as external_cmd = external_lookup.py So the minimal version of the working external lookup for returning the length of the field called "data" should look (with one caveat explained later) like this: transforms.conf: [test_lenlookup] external_cmd = lenlookup.py fields_list = data, length python.version = python3 And the lenlookup.py file itself: #!/usr/bin/env python3
import csv
import sys
def main():
infile = sys.stdin
outfile = sys.stdout
r = csv.DictReader(infile)
w = csv.DictWriter(outfile, fieldnames=["data","length"])
w.writeheader()
for result in r:
if result["data"]:
result["length"]=len(result["data"])
w.writerow(result)
main() Yes, it doesn't do any sanity checking or error handling but it does work for something like | makeresults | eval data="whatever" | lookup test_lenlookup data Of course this is a simple len() python function which in your case might or might not be what you need so the core functionality you might need to rewrite on your own. One important caveat. Even thought the spec file for transforms .conf says external_cmd = <string>
* Provides the command and arguments to invoke to perform a lookup. Use this
for external (or "scripted") lookups, where you interface with with an
external script rather than a lookup table.
* This string is parsed like a shell command.
* The first argument is expected to be a python script (or executable file)
located in $SPLUNK_HOME/etc/apps/<app_name>/bin.
* Presence of this field indicates that the lookup is external and command
based.
* Default: empty string I was unable to run my external lookup when the script was placed anywhere else than $SPLUNK_HOME/etc/system/bin. Judging from Answers history it seems to be some kind of a bug. EDIT: OK. I found it. It seems that for an external lookup to work you must give permissions to both the lookup definition (which you may as well do in WebUI) as well as to the script file itself (which you must do using the .meta file. So in this case you need something like this: [bin/lenlookup.py] access= read : [*] export = system [transforms/test_lenlookup] access = read : [*] export = system