Splunk Search

StreamingCommand block when input contains non-ascii character

jeffcui134
Engager

Environment:
splunk8.0
python3
splunk python SDK 1.6.11

When I write a customized command with python:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
from splunklib.searchcommands import \
    dispatch, StreamingCommand, Configuration, Option, validators
import splunk
@Configuration()   
class TestCommand(StreamingCommand):
    def stream(self, events):   
        for event in events:        
            yield event

dispatch(TestCommand, sys.argv, sys.stdin, sys.stdout, __name__)

This customized command always hang when input data has non-ascii character.
Such as: sourcetype=XXX| search url = "http://例子.卷筒纸" | testcommand

It seems splunkd crashed, since there has error log in splunkd.log:

10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: During handling of the above exception, another exception occurred:
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last):
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\testcommand.py", line 22, in
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: dispatch(TestCommand, sys.argv, sys.stdin, sys.stdout, name)
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\search_command.py", line 1118, in dispatch
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: command_class().process(argv, input_file, output_file)
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\search_command.py", line 435, in process
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self._process_protocol_v2(argv, ifile, ofile)
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\search_command.py", line 787, in _process_protocol_v2
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self.finish()
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\search_command.py", line 393, in finish
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self._record_writer.flush(finished=True)
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\internals.py", line 775, in flush
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self._write_chunk(metadata, self._buffer.getvalue())
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: File "C:\Program Files\Splunk\etc\apps\XXXX\bin\splunklib\searchcommands\internals.py", line 820, in _write_chunk
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: self._ofile.flush()
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: OSError: [Errno 22] Invalid argument
10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>

10-25-2019 16:34:42.808 +0800 ERROR ChunkedExternProcessor - stderr: OSError: [Errno 22] Invalid argument

There is no problem when I switch python version to 2.

Tags (1)

thellmann
Splunk Employee
Splunk Employee

Sorry for the thread necromancy, but we have solved issues with custom search commands hanging or crashing when sent multi-byte characters with an SDK update. This should be resolved in versions of the Splunk SDK for Python 1.6.15 and above. 

0 Karma

jeffcui134
Engager

This hang issue only reproduced when "chunked=true" in commands.conf
When I use search command protocol version 1, this issue doesn't replicate

[testcommand]
filename=testcommand.py
enableheader = true
outputheader = true
requires_srinfo = true
stderr_dest = message
supports_getinfo = true
supports_rawargs = true
supports_multivalues = true
0 Karma

zl0719
Engager

But version 1 performance is bottleneck. How to work out in search command protocol version 2?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...