StreamingCommand block when input contains non-ascii character

[jeffcui134]

Environment:
splunk8.0
python3
splunk python SDK 1.6.11

When I write a customized command with python:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
from splunklib.searchcommands import \
    dispatch, StreamingCommand, Configuration, Option, validators
import splunk
@Configuration()   
class TestCommand(StreamingCommand):
    def stream(self, events):   
        for event in events:        
            yield event

dispatch(TestCommand, sys.argv, sys.stdin, sys.stdout, __name__)

This customized command always hang when input data has non-ascii character.
Such as: sourcetype=XXX| search url = "http://例子.卷筒纸" | testcommand

It seems splunkd crashed, since there has error log in splunkd.log:

10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: During handling of the above exception, another exception occurred:
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last):
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\testcommand.py", line 22, in
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: dispatch(TestCommand, sys.argv, sys.stdin, sys.stdout, name)
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\search_command.py", line 1118, in dispatch
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: command_class().process(argv, input_file, output_file)
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\search_command.py", line 435, in process
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self._process_protocol_v2(argv, ifile, ofile)
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\search_command.py", line 787, in _process_protocol_v2
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self.finish()
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\search_command.py", line 393, in finish
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self._record_writer.flush(finished=True)
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\internals.py", line 775, in flush
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self._write_chunk(metadata, self._buffer.getvalue())
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\internals.py", line 820, in _write_chunk
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self._ofile.flush()
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: OSError: [Errno 22] Invalid argument
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>

10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: OSError: [Errno 22] Invalid argument

There is no problem when I switch python version to 2.

[thellmann]

Sorry for the thread necromancy, but we have solved issues with custom search commands hanging or crashing when sent multi-byte characters with an SDK update. This should be resolved in versions of the Splunk SDK for Python 1.6.15 and above. 

[jeffcui134]

This hang issue only reproduced when "chunked=true" in commands.conf
When I use search command protocol version 1, this issue doesn't replicate

[testcommand]
filename=testcommand.py
enableheader = true
outputheader = true
requires_srinfo = true
stderr_dest = message
supports_getinfo = true
supports_rawargs = true
supports_multivalues = true

[zl0719]

But version 1 performance is bottleneck. How to work out in search command protocol version 2?