All Apps and Add-ons

Index Cluster Custom Streaming Command No module named 'splunklib'

joepjisc
Path Finder

We have created a custom streaming command which does a computation based on two fields and adds a third with the result. 

This works fine on a single instance deployment, and on our search head cluster when used with `makeresults`, however, once the search heads offload the task to the indexing cluster each indexer throws the error 'No module named 'splunklib''.

The app has the custom command script in `./bin` and the `splunklib` directory from the SDK in `./lib/`.

 

 

#!/usr/bin/env python3
# other imports for computation
from os.path import join as realpath, dirname
from sys import path as syspath, argv, stdin, stdout
LibPath = realpath(dirname(realpath(__file__)) + '/../lib/')
syspath.insert(0, LibPath)
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option

@Configuration()
class OurCommand(StreamingCommand):
    # Option definitions
    def stream(self, events):
        for event in events:
            # computation logic
            yield event

dispatch(SubnetOfCommand, argv, stdin, stdout, __name__)

 

 

The full error from the log for each indexer is:

01-20-2021 09:37:33.390 INFO ChunkedExternProcessor - Running process: /opt/splunk/bin/python3.7 /opt/splunk/var/run/searchpeers/7E2EF370-95B8-474E-B6F6-47F96425213C-1611135318/apps/TA-ourcommand/bin/OurCommand.py
01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last):
01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: File "/opt/splunk/var/run/searchpeers/7E2EF370-95B8-474E-B6F6-47F96425213C-1611135318/apps/TA-ourcommand/bin/OurCommand", line 16, in <module>
01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option
01-20-2021 09:37:33.437 ERROR ChunkedExternProcessor - stderr: ModuleNotFoundError: No module named 'splunklib'
01-20-2021 09:37:33.440 ERROR ChunkedExternProcessor - EOF while attempting to read transport header read_size=0
01-20-2021 09:37:33.440 ERROR ChunkedExternProcessor - Error in 'ourcommand' command: External search command exited unexpectedly with non-zero error code 1.
01-20-2021 09:37:33.445 ERROR SearchPipelineExecutor - sid:remote_searchead.fqdn__am9lLnBpdHQ_am9lLnBpdHQ_amlzY19jc2lydF9jdGk__search3_1611135450.4456_4BFB5A9E-3ADF-405B-B424-200C91CD6F72 Streamed search execute failed because: Error in 'ourcommand' command: External search command exited unexpectedly with non-zero error code 1..

 

It appears the custom search command is being copied to a temporary folder under `/opt/splunk/var/run/`; so wonder if the lib directory is being missed.

 

Any suggestions on how to resolve this would be really appreicated.

Labels (1)
0 Karma

to4kawa
Ultra Champion

/Applications/Splunk/lib/python3.7/site-packages

lrwxr-xr-x    1 XXXXX  XXXXX     52 12 31 10:53 splunklib -> /opt/anaconda3/lib/python3.7/site-packages/splunklib

 

I experienced the same problem.
I managed to set up a soft link.

joepjisc
Path Finder

Thanks for the quick reply, presumably this doesn't have any adverse impact on upgrading Splunk or anything like that.

If I've understood correctly, the below would need to be done manually on each indexer node, is that correct?

cp -r /opt/splunk/etc/apps/TA-ourcommand/lib/splunklib /opt/
chown -R splunk: /opt/splunklib/
cd /opt/splunk/lib/python3.7/site-packages/
ln -s /opt/splunklib/ splunklib

 

0 Karma

to4kawa
Ultra Champion

I think it's a search head also. 

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...