Can't use stats with custom streaming searchcommand

[wesleya]

I have a custom search command that extracts a domain name from a url string field you specify into a new "domain" field. This works fine on a dev cluster we have setup (3 search heads, 2 indexers). For example this returns expected results:

index=main
| table _time url
| mycustomcommand field_in=url

but adding stats command at the end of the search causes the search to fail with the following error:

index=main
| table _time url
| mycustomcommmand field_in=url
| stats count by domain

2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.
[ip-{indexer_1_ip}] Streamed search execute failed because: Error in 'mycustomcommmand' command: External search command exited unexpectedly with non-zero error code 1..
[ip-{indexer_2_ip}] Streamed search execute failed because: Error in 'mycustomcommmand' command: External search command exited unexpectedly with non-zero error code 1..

Running the search directly on the indexer returns 0 results, because we don't have the url field extraction there. But there are no errors.

My questions are

1. Where can I find the reason for the failure? I can't seem to find what the actual error is anywhere in the search.log.
2. Any ideas about what's going on here, or documentation that may help?

[richgalloway]

Try appending | noop log_DEBUG=* to the search.  Then check the search log for debug messages that may help determine the cause of the error.

[wesleya · Accepted solution]

Thank you for the help! This led me to figure out I was only looking at the logs for the search head.  The search was streamed to indexers when using the stats command, and those indexer search.log files can be found through the job inspector under the Search Job Properties link.

The script errors found there (ImportError: No module named {mylib}) led me to this answer which explains the problem nicely: https://community.splunk.com/t5/Developing-for-Splunk-Enterprise/Custom-streaming-search-command-err...