Index Cluster Custom Streaming Command No module named 'splunklib'

[joepjisc]

We have created a custom streaming command which does a computation based on two fields and adds a third with the result. 

This works fine on a single instance deployment, and on our search head cluster when used with `makeresults`, however, once the search heads offload the task to the indexing cluster each indexer throws the error 'No module named 'splunklib''.

The app has the custom command script in `./bin` and the `splunklib` directory from the SDK in `./lib/`.

#!/usr/bin/env python3
# other imports for computation
from os.path import join as realpath, dirname
from sys import path as syspath, argv, stdin, stdout
LibPath = realpath(dirname(realpath(__file__)) + '/../lib/')
syspath.insert(0, LibPath)
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option

@Configuration()
class OurCommand(StreamingCommand):
    # Option definitions
    def stream(self, events):
        for event in events:
            # computation logic
            yield event

dispatch(SubnetOfCommand, argv, stdin, stdout, __name__)

The full error from the log for each indexer is:

01-20-2021 09:37:33.390 INFO ChunkedExternProcessor - Running process: /opt/splunk/bin/python3.7 /opt/splunk/var/run/searchpeers/7E2EF370-95B8-474E-B6F6-47F96425213C-1611135318/apps/TA-ourcommand/bin/OurCommand.py
01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last):
01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: File "/opt/splunk/var/run/searchpeers/7E2EF370-95B8-474E-B6F6-47F96425213C-1611135318/apps/TA-ourcommand/bin/OurCommand", line 16, in <module>
01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option
01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: ModuleNotFoundError: No module named 'splunklib'
01-20-2021 09:37:33.440 ERROR ChunkedExternProcessor - EOF while attempting to read transport header read_size=0
01-20-2021 09:37:33.440 ERROR ChunkedExternProcessor - Error in 'ourcommand' command: External search command exited unexpectedly with non-zero error code 1.
01-20-2021 09:37:33.445 ERROR SearchPipelineExecutor - sid:remote_searchead.fqdn__am9lLnBpdHQ_am9lLnBpdHQ_amlzY19jc2lydF9jdGk__search3_1611135450.4456_4BFB5A9E-3ADF-405B-B424-200C91CD6F72 Streamed search execute failed because: Error in 'ourcommand' command: External search command exited unexpectedly with non-zero error code 1..

It appears the custom search command is being copied to a temporary folder under `/opt/splunk/var/run/`; so wonder if the lib directory is being missed.

Any suggestions on how to resolve this would be really appreicated.

[to4kawa]

/Applications/Splunk/lib/python3.7/site-packages

lrwxr-xr-x    1 XXXXX  XXXXX     52 12 31 10:53 splunklib -> /opt/anaconda3/lib/python3.7/site-packages/splunklib

I experienced the same problem.
I managed to set up a soft link.

[joepjisc]

Thanks for the quick reply, presumably this doesn't have any adverse impact on upgrading Splunk or anything like that.

If I've understood correctly, the below would need to be done manually on each indexer node, is that correct?

cp -r /opt/splunk/etc/apps/TA-ourcommand/lib/splunklib /opt/
chown -R splunk: /opt/splunklib/
cd /opt/splunk/lib/python3.7/site-packages/
ln -s /opt/splunklib/ splunklib

[to4kawa]

I think it's a search head also.