Splunk Search
Highlighted

Any advice for troubleshooting scripted lookups?

SplunkTrust
SplunkTrust

I'm trying to create a scripted lookup and I'm finding it a little frustrating because any time there's a python exception the lookup just throws an error in the UI

Script for lookup table 'call_quality' returned error code 1. Results may be incorrect.

and nothing gets written to splunkd.log nor python.log. So I have to figure out what the python exception is on my own, and in practice this is a pretty painful way to develop something.

Am I just missing something obvious? Thanks.

Highlighted

Re: Any advice for troubleshooting scripted lookups?

Builder

Modify etc/log.cfg for the following blocks:

#
# define splunk python logging properties
#
# logging classes are defined by a logging declaration at the log of each
# file.
#
#       splunk
#       splunk.appserver
#       splunk.search
#
[python]
splunk = INFO
splunk.appserver = INFO
splunk.appserver.controllers = INFO
splunk.appserver.lib = WARN

Setting them to DEBUG should echo everything from each script inside the python.log file.

0 Karma
Highlighted

Re: Any advice for troubleshooting scripted lookups?

SplunkTrust
SplunkTrust

No such luck. Nothing comes out to python.log nor any other splunk log. Incidentally even when set to DEBUG, python.log remains completely empty.

0 Karma
Highlighted

Re: Any advice for troubleshooting scripted lookups?

SplunkTrust
SplunkTrust

ok. I tried setting everything in [python] to DEBUG and restarting but there still aren't any errors coming out.

0 Karma
Highlighted

Re: Any advice for troubleshooting scripted lookups?

Builder

but the calls to the lookup are being recorded in the python.log?

0 Karma
Highlighted

Re: Any advice for troubleshooting scripted lookups?

SplunkTrust
SplunkTrust

nope. python.log remains completely empty. permissions on all the logs are the same fwiw.

0 Karma
Highlighted

Re: Any advice for troubleshooting scripted lookups?

Builder

I'm not sure why python.log isn't being updated. Might be worth importing logger into your script and use a try clause to catch your own exception. Then you could also log the updates / returns from when the script is ran.

0 Karma
Highlighted

Re: Any advice for troubleshooting scripted lookups?

SplunkTrust
SplunkTrust

my interpretation is that on line 25 of runScript.py, i should change "logging.INFO or logging.ERROR" to "logging.DEBUG or logging.INFO or logging.ERROR". However I tried that and restarted and it had no effect. python.log is still empty.

0 Karma
Highlighted

Re: Any advice for troubleshooting scripted lookups?

Builder

I tried changing the rest_handler.py to info and debug and can't get any of my custom rest calls to output anything to python.log. I'm very confused on why. I'm hoping someone else can post why in here, or this may be a bug. I know changing the base log.cfg as discussed below used to record everything python called.

0 Karma
Highlighted

Re: Any advice for troubleshooting scripted lookups?

Hi Nick,
have you tried running it from splunk's cmd line?

splunk cmd python <pathtoyourscript> 

Still no luck? Isn't it possible to force your script's logging to happen somewhere else, something like:

import logging 

LOG_FILENAME = '/tmp/mylookupscript.log'
LOG_FORMAT = "[%(asctime)s] %(name)s %(levelname)s: %(message)s"
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG,format=LOG_FORMAT)
logger = logging.getLogger('mylookupscript')
0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.