External python search - How do I resolve Error "Failed to parse transport header"?

[kepffr]

Hi guys,

I'm trying to write a very simple external python search but it's just not working.

I get the following error message in search_messages.log:

06-15-2021 09:44:22.543 +0200 ERROR SearchMessages - orig_component="script" app="search" sid="1623743052.198909" message_key="EXTERN:SCRIPT_NONZERO_RETURN" message=External search command 'pyTest' returned error code 1. Script output = "chunked 1.0,241,0\n{"inspector":{"messages":[["ERROR","RuntimeError at \"D:\\Splunk\\etc\\apps\\pyTest\\bin\\splunklib\\searchcommands\\search_command.py\", line 884 : Failed to parse transport header: b'splunkVersion:8.2.0\\n'"]]},"finished":true}".

It says message_key="EXTERN:SCRIPT_NONZERO_RETURN" and "Failed to parse transport header".

 This is how I call the script in a splunk search:

| makeresults 1 | eval something="just_a_value" | script pyTest

or

| script pyTest

This is my commands.conf:

[pyTest]
python.version = python3
chunked = true
filename = pyTest.py

This is my code:

#!/usr/bin/python3
import os, sys

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
import splunk.Intersplunk
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration

@Configuration()
class pyTest(StreamingCommand):
    def stream(self, events):
        for event in events:
            event['nothing'] = 'world'
            yield event

dispatch(pyTest, sys.argv, sys.stdin, sys.stdout, __name__)

I have also tried to replace \r\n with \n in the code but that didn't help. What am I doing wrong here?

[kepffr]

Hi guys. Any ideas?

[bpna]

I solved it! The solution reflects SO poorly on Splunk.

I ran my command with "--debug" and got this output:

Command list_entities appears to be statically configured for search command protocol version 1 and static configuration is unsupported by splunklib.searchcommands. Please ensure that default/commands.conf contains this stanza: [generatetext] filename = generatetext.py enableheader = true outputheader = true requires_srinfo = true supports_getinfo = true

If you add those settings to your commands.conf it should work

Splunk's OWN DOCS AND EXAMPLES IN MULTIPLE PLACES say "chunked = true" specifies search command protocol version 2

Like here https://dev.splunk.com/enterprise/docs/devtools/customsearchcommands/createcustomsearchcmd#Register-...

And here https://docs.splunk.com/Documentation/ITSI/4.12.0/Configure/commands.conf

And here https://github.com/splunk/splunk-sdk-python/blob/master/examples/searchcommands_app/package/default/...

Wow.

[highsplunker]

Thank you! Worked OK!

I faced the same error. You helped a lot! 🙂

[bpna]

I'm having the same issue.

Actually after pulling my hair out over it, copied the generatetext command example from Splunk SDK for Python GitHub repo into my instance https://github.com/splunk/splunk-sdk-python/blob/master/examples/searchcommands_app/package/bin/gene...

Copied default/commands.conf. Setup splunklib directory.

Splunk v8.2.2, should play nicely with python3

Still failing with this error. Did you ever solve it?

[ohbuckeyeio]

FWIW, I had this same problem and fixed it by changing my OS command from subprocess.run() to subprocess.Popen.  Not sure why, but it seem's Splunk's implementation of Python 3.7 has issues with subprocess.run() and streaming commands.

This allowed me to use 

chunked = true

without any problems. 

[bpna]

That's interesting, thanks for the info. Helpful to know it's an issue with Splunk's usage of python internally.

I was able to solve it by editing commands.conf to the following:

[list_entities]
filename = list_entities.py
enableheader = true
outputheader = true
requires_srinfo = true
supports_getinfo = true
supports_multivalues = true
supports_rawargs = true
python.version = python3