Why does my external scripted lookup give me an error code 1?

[reswob4]

So I created a way to use Mark Baggett's freq_server script as a lookup and blogged about it here (http://shadowtrackers.net/blog/get-your-freq-on-in-splunk).

But, that is not the point of this forum post.

I had based my blog post on my setup at home. I was running a single Splunk server and making the URL connection to a separate Linux server running in a vm on the same box as the Splunk server. So, when I got to work and tried to implement the same lookup, I was surprised and frustrated when it didn't work right out of the box. On the small Splunk instance where I implemented the lookup, I have a single search head and two Indexers. When I ran the lookup search:

index=dns | rename query as domain | lookup freqserver domain

I received the following error:

    2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.   

        [INDEX01] Script for lookup table 'freqserver' returned error code 1. Results may be incorrect.
        [INDEX02] Script for lookup table 'freqserver' returned error code 1. Results may be incorrect.

[reswob4 · Accepted solution]

My first thought was WTH?

But here are some of the things I tried:

https://answers.splunk.com/answers/369810/python-alert-script-fails-and-i-cant-see-errors-in.html
https://answers.splunk.com/answers/559456/script-for-lookup-table-whois-returned-error-code.html

But ultimately the information in the error clued me in. The linux server at work where I was running the frequency analysis server had iptables running and I had configured it to allow my SH to connect on the exact port the frequency analysis server was listening on. But what I didn't know, and isn't mentioned here (https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Configureexternallookups), is that the script is actually run from the search peers (which in my case were the indexers). Since the search peers were not permitted to connect to the frequency analysis server per the iptables rules, the script was blocked from connecting and thus it failed.

Once I opened up iptables to allow the search peers access to the frequency analysis server, everything worked.

Maybe someone from Splunk can better explain, but apparently the script is copied from the SH to one of the following locations (or both) on each search peer and run from there:

$SPLUNK_HOME$/var/run/searchpeers/<servername>-#########/searchscripts/

or

$SPLUNK_HOME$/var/run/searchpeers/<servername>-#########/system/bin/

Hopefully this helps someone else troubleshoot their scripted lookups....