Cannot get external lookup to work

[vincesesto]

I am trying to set up a lookup in my test environment to hopefully push out to production. I have created an app and set up the transforms and props with the following details(By the way, a clog is an internal process we use at my work):
translforms.conf
[extraClogDetails]
external_cmd = clogvalues.py date popcode pages
external_type = python
fields_list = date, popcode, pages

props.conf
[clogsreceived]
LOOKUP-pages = extraClogDetails date popcode pages OUTPUT pages

I have created my external lookup script and when I run the script catting in an csv file, all seems fine:
(Running Script from command line) python clogvalues.py date popcode pages < /tmp/test.csv
(Output)
date,popcode,pages
2012-08-09,us_isp_nz3,42331
2012-08-09,us_isp_nz5,44694

(entries in test.csv)
date,popcode,pages
"2012-08-09","us_isp_nz3",
"2012-08-09","us_isp_nz5",

When I try to do a lookup from the command line, I get no output for pages:
./splunk search "sourcetype=clogsreceived popcode="*callplus2_27" | lookup extraClogDetails date popcode pages OUTPUT pages"
date=2012-08-09 00:00:00+10:00 popcode=us_isp_nz3
date=2012-08-09 00:00:00+10:00 popcode=us_isp_nz5

I am not sure what I have missed and I am not getting any errors from splunk. Is there anything further that I can check, or is there a better debug mode that will show me where I am going wrong.

Any assistance would be appreciated.

Regards Vince

[vincesesto]

Hello All,
Just letting you know that this issue was caused by splunk not working with external python modules. Specifically in this case the psycopg2 module used to connect with postgres databases. I was not able to load the module into the Splunk python environment so needed to resort to creating a wrapper script as per ticket:
http://splunk-base.splunk.com/answers/10839/scripted-lookup-script-doesnt-work-with-splunk-python-ve...

For those interested the wrapper script is very simple, and takes the command line arguements, and then sends them to your lookup script as per below. This script is the one referenced in the transfors.conf for the external_cmd lookup configs:

import os, sys
real_script = "/opt/splunk/etc/apps/BMSDBAccess/bin/clogvalues.py"
python_executable = "/usr/bin/python"
os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:])

That is all there is to it, but it does slow things down quite a bit having to go through this extra wrapper script to get the job done...I would be great for splunk to either include this module into their environment or resolve this issue as it was the last thing I was expecting.

Regards Vince

[stefano_guidoba]

Vince can you please post your clogvalues.py script?
Regards,
Stefano

[Ayn · Accepted solution]

You shouldn't be supplying the OUTPUT field as an input field as well. You have this in your search:

lookup extraClogDetails date popcode pages OUTPUT pages

But it needs to be

lookup extraClogDetails date popcode OUTPUT pages

so that pages is only in the output, not on both sides.

[vincesesto]

Hey Ayn,
Just want to thank you for helping me with this. I finally got this working as I was using the python psycopg2 library to connect to the database...Turns out this will not work with the splunk python. I have set up a wrapper script to get this working finally.

Thanks again.

Regards, Vince

[Ayn]

It certainly looks that way. Something in your lookup script is throwing an error. What I've done when I've developed dynamic lookups that I wanted to troubleshoot in the past is to redirect stderr to some file so I can see what errors the lookup is throwing.

f = open("/path/to/some/file","w")
sys.stderr = f

[vincesesto]

Hello,
So I am no longer using the OUTPUT field, but now I am still not able to get my pages values filled. I am getting the following output:
splunk@crunchbang:~/bin$ ./splunk search "sourcetype="clogsreceived" popcode="us_isp_au1" | lookup clog_details date popcode OUTPUT pages"
ERROR: Script for lookup table 'clog_details' returned error code 1. Results may be incorrect.
date=2012-08-09 popcode=us_isp_au1 up_page_limit=201348.0 low_page_limit=134232.0
date=2012-08-08 popcode=us_isp_au1 up_page_limit=204788.4 low_page_limit=136525.6

The ERROR seems to be killing my lookup.
Vince

[Ayn]

Splunk will look at the fields_list in your lookup definition and send those fields to the lookup script, regardless of which fields you specify yourself in the actual lookup command. So, you don't need to worry about that yourself in your script.

[vincesesto]

Hey Ayn,

Thanks for your response. I understand what you are saying but do you know how that will work with my python script. In the examples provided, you are supposed to give a header that will include the empty field, in this case I have the header of:
date, popcode, pages

And my understanding is that splunk will fill in the blank page values...Will I need to amend my script to no longer look for the "pages" field in the header?

Regards,

Vince