I'm new to Splunk, and I have some basic questions about custom search commands. I wrote a Python script and put it in
$splunk_home\etc\apps\search\bin, and I added a stanza to
$splunk_home\etc\local\commands.conf and restarted Splunk. When I try to pipe a search to the custom command, I get a message saying "External search command 'foo' failed with error code 1." I don't know what error code 1 is, and I can't find any useful details in any of the files in
var\log\splunk--the search string shows up in searches.log, but nothing else looks relevant. It's possible I have a straightforward problem (like a syntax error or the files in the wrong place), but how do I go about pinpointing it? Can I turn up the error reporting in the web interface? What log files are supposed to be useful here?
Unfortunately there isn't a whole lot of logging going on about errors in custom search scripts. Id's suggests setting up your own top-level log handled in your script to capture and log any exceptions yourself. You can throw the logs into the
$SPLUNK_HOME/var/log/splunk/ directory and then the events will be available with a
index=_internal search, or just put them in
/tmp/ if you want something quick and dirty.
An example is posted here:
I did a quick test and if my python search command had any typos in it would report the same "error code 1" like you saw. (I just threw in an extra line with the word "BLAH". This raises a NameError and python returns with an exit code of 1.) Like you saw, I couldn't find anything in the logs that showed me the error specifically.
So catch top-level exceptions, and consider putting all of your logic into a top-level function. This gives the ability to use tools like
pychecker or even write unit tests around a search command core logic and do test independently of a splunk search.
@Lowell, thanks for the reply. Since I posted this, I figured out how to run the bundled Python interpreter (.\splunk cmd python), and I tested non-Splunk subsections of my script successfully. I tried to set up custom logging but didn't have much luck; I'll give it another shot.
The only parts of my script I couldn't test directly were the calls to Intersplunk. Is there a way to provide input/output from the command line, e.g. a small specially formatted text file as a dummy result?
Finally came back to this. I ran
splunk cmd python .\myscript.py searchargs
Once I fixed the backslash issue, the custom log worked as described above.