We have created a custom streaming command which does a computation based on two fields and adds a third with the result.
This works fine on a single instance deployment, and on our search head cluster when used with `makeresults`, however, once the search heads offload the task to the indexing cluster each indexer throws the error 'No module named 'splunklib''.
The app has the custom command script in `./bin` and the `splunklib` directory from the SDK in `./lib/`.
# other imports for computation
from os.path import join as realpath, dirname
from sys import path as syspath, argv, stdin, stdout
LibPath = realpath(dirname(realpath(__file__)) + '/../lib/')
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option
# Option definitions
def stream(self, events):
for event in events:
# computation logic
dispatch(SubnetOfCommand, argv, stdin, stdout, __name__)
The full error from the log for each indexer is:
01-20-2021 09:37:33.390 INFO ChunkedExternProcessor - Running process: /opt/splunk/bin/python3.7 /opt/splunk/var/run/searchpeers/7E2EF370-95B8-474E-B6F6-47F96425213C-1611135318/apps/TA-ourcommand/bin/OurCommand.py 01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last): 01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: File "/opt/splunk/var/run/searchpeers/7E2EF370-95B8-474E-B6F6-47F96425213C-1611135318/apps/TA-ourcommand/bin/OurCommand", line 16, in <module> 01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option 01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: ModuleNotFoundError: No module named 'splunklib' 01-20-2021 09:37:33.440 ERROR ChunkedExternProcessor - EOF while attempting to read transport header read_size=0 01-20-2021 09:37:33.440 ERROR ChunkedExternProcessor - Error in 'ourcommand' command: External search command exited unexpectedly with non-zero error code 1. 01-20-2021 09:37:33.445 ERROR SearchPipelineExecutor - sid:remote_searchead.fqdn__am9lLnBpdHQ_am9lLnBpdHQ_amlzY19jc2lydF9jdGk__search3_1611135450.4456_4BFB5A9E-3ADF-405B-B424-200C91CD6F72 Streamed search execute failed because: Error in 'ourcommand' command: External search command exited unexpectedly with non-zero error code 1..
It appears the custom search command is being copied to a temporary folder under `/opt/splunk/var/run/`; so wonder if the lib directory is being missed.
Any suggestions on how to resolve this would be really appreicated.