My first thought was WTH?
But here are some of the things I tried:
https://answers.splunk.com/answers/369810/python-alert-script-fails-and-i-cant-see-errors-in.html
https://answers.splunk.com/answers/559456/script-for-lookup-table-whois-returned-error-code.html
But ultimately the information in the error clued me in. The linux server at work where I was running the frequency analysis server had iptables running and I had configured it to allow my SH to connect on the exact port the frequency analysis server was listening on. But what I didn't know, and isn't mentioned here (https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Configureexternallookups), is that the script is actually run from the search peers (which in my case were the indexers). Since the search peers were not permitted to connect to the frequency analysis server per the iptables rules, the script was blocked from connecting and thus it failed.
Once I opened up iptables to allow the search peers access to the frequency analysis server, everything worked.
Maybe someone from Splunk can better explain, but apparently the script is copied from the SH to one of the following locations (or both) on each search peer and run from there:
$SPLUNK_HOME$/var/run/searchpeers/<servername>-#########/searchscripts/
or
$SPLUNK_HOME$/var/run/searchpeers/<servername>-#########/system/bin/
Hopefully this helps someone else troubleshoot their scripted lookups....
... View more