Splunk Search

Cannot get external lookup to work

vincesesto
Communicator

I am trying to set up a lookup in my test environment to hopefully push out to production. I have created an app and set up the transforms and props with the following details(By the way, a clog is an internal process we use at my work):
translforms.conf
[extraClogDetails]
external_cmd = clogvalues.py date popcode pages
external_type = python
fields_list = date, popcode, pages

props.conf
[clogsreceived]
LOOKUP-pages = extraClogDetails date popcode pages OUTPUT pages

I have created my external lookup script and when I run the script catting in an csv file, all seems fine:
(Running Script from command line) python clogvalues.py date popcode pages < /tmp/test.csv
(Output)
date,popcode,pages
2012-08-09,us_isp_nz3,42331
2012-08-09,us_isp_nz5,44694

(entries in test.csv)
date,popcode,pages
"2012-08-09","us_isp_nz3",
"2012-08-09","us_isp_nz5",

When I try to do a lookup from the command line, I get no output for pages:
./splunk search "sourcetype=clogsreceived popcode="*callplus2_27" | lookup extraClogDetails date popcode pages OUTPUT pages"
date=2012-08-09 00:00:00+10:00 popcode=us_isp_nz3
date=2012-08-09 00:00:00+10:00 popcode=us_isp_nz5

I am not sure what I have missed and I am not getting any errors from splunk. Is there anything further that I can check, or is there a better debug mode that will show me where I am going wrong.

Any assistance would be appreciated.

Regards Vince

0 Karma
1 Solution

Ayn
Legend

You shouldn't be supplying the OUTPUT field as an input field as well. You have this in your search:

lookup extraClogDetails date popcode pages OUTPUT pages

But it needs to be

lookup extraClogDetails date popcode OUTPUT pages

so that pages is only in the output, not on both sides.

View solution in original post

vincesesto
Communicator

Hello All,
Just letting you know that this issue was caused by splunk not working with external python modules. Specifically in this case the psycopg2 module used to connect with postgres databases. I was not able to load the module into the Splunk python environment so needed to resort to creating a wrapper script as per ticket:
http://splunk-base.splunk.com/answers/10839/scripted-lookup-script-doesnt-work-with-splunk-python-ve...

For those interested the wrapper script is very simple, and takes the command line arguements, and then sends them to your lookup script as per below. This script is the one referenced in the transfors.conf for the external_cmd lookup configs:

import os, sys
real_script = "/opt/splunk/etc/apps/BMSDBAccess/bin/clogvalues.py"
python_executable = "/usr/bin/python"
os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:])

That is all there is to it, but it does slow things down quite a bit having to go through this extra wrapper script to get the job done...I would be great for splunk to either include this module into their environment or resolve this issue as it was the last thing I was expecting.

Regards Vince

stefano_guidoba
Communicator

Vince can you please post your clogvalues.py script?
Regards,
Stefano

0 Karma

Ayn
Legend

You shouldn't be supplying the OUTPUT field as an input field as well. You have this in your search:

lookup extraClogDetails date popcode pages OUTPUT pages

But it needs to be

lookup extraClogDetails date popcode OUTPUT pages

so that pages is only in the output, not on both sides.

vincesesto
Communicator

Hey Ayn,
Just want to thank you for helping me with this. I finally got this working as I was using the python psycopg2 library to connect to the database...Turns out this will not work with the splunk python. I have set up a wrapper script to get this working finally.

Thanks again.

Regards, Vince

0 Karma

Ayn
Legend

It certainly looks that way. Something in your lookup script is throwing an error. What I've done when I've developed dynamic lookups that I wanted to troubleshoot in the past is to redirect stderr to some file so I can see what errors the lookup is throwing.

f = open("/path/to/some/file","w")
sys.stderr = f

vincesesto
Communicator

Hello,
So I am no longer using the OUTPUT field, but now I am still not able to get my pages values filled. I am getting the following output:
splunk@crunchbang:~/bin$ ./splunk search "sourcetype="clogsreceived" popcode="us_isp_au1" | lookup clog_details date popcode OUTPUT pages"
ERROR: Script for lookup table 'clog_details' returned error code 1. Results may be incorrect.
date=2012-08-09 popcode=us_isp_au1 up_page_limit=201348.0 low_page_limit=134232.0
date=2012-08-08 popcode=us_isp_au1 up_page_limit=204788.4 low_page_limit=136525.6

The ERROR seems to be killing my lookup.
Vince

0 Karma

Ayn
Legend

Splunk will look at the fields_list in your lookup definition and send those fields to the lookup script, regardless of which fields you specify yourself in the actual lookup command. So, you don't need to worry about that yourself in your script.

0 Karma

vincesesto
Communicator

Hey Ayn,

Thanks for your response. I understand what you are saying but do you know how that will work with my python script. In the examples provided, you are supposed to give a header that will include the empty field, in this case I have the header of:
date, popcode, pages

And my understanding is that splunk will fill in the blank page values...Will I need to amend my script to no longer look for the "pages" field in the header?

Regards,

Vince

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...