A bytes like object is required, not string - Splunk DB Connect

[hakon_nor]

Hi,

When I try to set the JRE Installation path(java_home) during configuration of Splunk DB connect 3.2.0 I get the following error; "A bytes like object is required, not string". I'm currently using Splunk Enterprise 8.0.1, with python 3.7 set as the default python version. This error only occurs when I use python 3.7, using python 2.7 removes the problem.

The whole problem seems to originate from how the different python versions handles strings, as described by Splunk(I dont have enough karma to post links so ill just quote it):

"In Python 2, a native, default string is a binary or "bytes" string. In Python 3, a native string is instead a unicode string. Python 3 is strict about not mixing different types of strings. Often, an explicit conversion is needed or a runtime error is produced.

In Python 2, default and binary strings shared the same type but in Python 3, you now must keep the two string types distinctly separated. Binary data should not be stored in a default string since Python 3 defaults to unicode strings, not binary."

So my question is how do I convert the string to a bytes like object?

[romankris]

The reason for this error is that in Python 3, strings are Unicode, but when transmitting on the network, the data needs to be bytes instead. We can convert bytes to string using bytes class decode() instance method, So you need to decode the bytes object to produce a string. In Python 3 , the default encoding is "utf-8" , so you can use directly:

b"python byte to string".decode("utf-8")

Python makes a clear distinction between bytes and strings . Bytes objects contain raw data — a sequence of octets — whereas strings are Unicode sequences . Conversion between these two types is explicit: you encode a string to get bytes, specifying an encoding (which defaults to UTF-8); and you decode bytes to get a string. Clients of these functions should be aware that such conversions may fail, and should consider how failures are handled.

[hakon_nor · Accepted solution]

Posting the solution to the original question given in the comments for visibility:

The solution was to change jre_validator.py under splunk_app_db_connect/bin/dbx2 to take into account the different string types. Splunk docs provided the solution, replace the file with the following code:

#! /usr/bin/env python
import os
from .dbx_logger import logger
import threading
import subprocess
import traceback
import shlex
from . import C
import sys
from six import string_types

class Command(object):
    """
    Enables to run subprocess commands in a different thread with TIMEOUT option.

    Based on jcollado's solution:
    http://stackoverflow.com/questions/1191374/subprocess-with-timeout/4825933#4825933
    """
    command = None
    process = None
    status = None
    output, error = '', ''

    def __init__(self, command):
        # string_types is a basestring() in Python 2 and str in Python 3
        if isinstance(command, string_types):
            # split the command using shell like syntax
            command = shlex.split(command)
        self.command = command


    def run(self, timeout=None, **kwargs):
        """ Run a command then return: (status, output, error). """
        def target(**kwargs):
            try:
                self.process = subprocess.Popen(self.command, **kwargs)
                self.output, self.error = self.process.communicate()
                self.status = self.process.returncode
            except Exception as ex:
                self.error = traceback.format_exception_only(type(ex), ex)
                self.status = -1
        # default stdout and stderr
        if 'stdout' not in kwargs:
            kwargs['stdout'] = subprocess.PIPE
        if 'stderr' not in kwargs:
            kwargs['stderr'] = subprocess.PIPE
        # thread
        thread = threading.Thread(target=target, kwargs=kwargs)
        thread.start()
        thread.join(timeout)
        if thread.is_alive():
            self.process.terminate()
            thread.join()
        return self.status, self.output, self.error


def matchesAny(line, values):
    return line.lstrip().startswith(tuple(values))

def validateJRE(javaCmd):
    attrs = {}
    reason = ""
    operation = "validate java command: {}.".format(javaCmd)

    if os.name == "nt":
        jreInfoCmd = '"%s" %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    else:
        jreInfoCmd = '%s %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    c = Command(jreInfoCmd)
    retval, output, error = c.run(10)
    if sys.version_info[0] == 3:
        output = output.decode('utf-8')
        error = error.decode('utf-8')

    if retval != 0:
        reason = error
        logger.debug(error)
        # smartly parse the error if we can.
        for line in error:
            if line.startswith("OSError:") or line.startswith("WindowsError"):
                reason = line
        isValid = False
        logger.critical(reason)
    else:
        pairs = [ line.lstrip().split(' = ', 2) for line in error.splitlines() if matchesAny(line, C.JRE_WANTED_KEYS)]

        for pair in pairs:
            k, v = pair
            attrs[k] = v

        version = attrs.get(C.JRE_VERSION_KEY, '')
        vendor = attrs.get(C.JRE_VENDOR_KEY, '')
        vm = attrs.get(C.JRE_VM_KEY, '')

        isValid = (version == C.JRE_WANT_VERSION and (vm.startswith(C.JRE_WANT_VM_ORACLEJDK) or vm.startswith(C.JRE_WANT_VM_OPENJDK)))
        if not isValid:
            reason = {"message": "Unsupported JRE detected",
                      "jre_using": "Using %s JRE version %s, %s" % (vendor, version, vm),
                      "jre_need": "Need Oracle Corporation JRE version 1.8 or OpenSDK 1.8"}
            logger.critical(reason)

    details = [str(reason), operation]

    return isValid, " ".join(details)

if os.name == 'nt':
    JAVA_DEPENDENCIES = [os.path.join("bin", "java.exe"),
                         os.path.join("bin", "keytool.exe")]

else:
    JAVA_DEPENDENCIES = [os.path.join("bin", "java"),
                         os.path.join("bin", "keytool")]

def checkDependencies(javaHome):
    reason = ""
    for dep in JAVA_DEPENDENCIES:
        fullPath = os.path.join(javaHome, dep)
        if not os.path.exists(fullPath):
            reason = "Missing JRE dependency: %s" % fullPath
            logger.critical(reason)
            return False, reason
    return True, reason         

[ips_mandar]

Hi @hakon_nor,
I am also getting error in python and I tried replacing code with yours in jre_validator.py but due to which I am getting error as-<?xml version="1.0" encoding="UTF-8"?> <response> <messages> <msg type="WARN">An exception was thrown while dispatching the python script handler. </msg> </messages> </response>
and unable to load any option in splunk db connect. Kindly help is there any other things need to be change?

[hakon_nor]

The error is probably caused by an indentation error in the provided script. Seems like the code sample tool used in these comments adds another space before each line of code.

[ips_mandar]

yes thank you

[afx]

Thanks, the Java option did the trick. It created the file jars/server.vmopts:

-Ddw.server.applicationConnectors[0].port=9998

And now it all works!
thx
afx

[hakon_nor]

Fixed it. The solution was to change jre_validator.py under splunk_app_db_connect/bin/dbx2 to take into account the different string types. Splunk docs provided the solution, replace the file with the following code:

#! /usr/bin/env python
import os
from .dbx_logger import logger
import threading
import subprocess
import traceback
import shlex
from . import C
import sys
from six import string_types

class Command(object):
    """
    Enables to run subprocess commands in a different thread with TIMEOUT option.

    Based on jcollado's solution:
    http://stackoverflow.com/questions/1191374/subprocess-with-timeout/4825933#4825933
    """
    command = None
    process = None
    status = None
    output, error = '', ''

    def __init__(self, command):
        # string_types is a basestring() in Python 2 and str in Python 3
        if isinstance(command, string_types):
            # split the command using shell like syntax
            command = shlex.split(command)
        self.command = command


    def run(self, timeout=None, **kwargs):
        """ Run a command then return: (status, output, error). """
        def target(**kwargs):
            try:
                self.process = subprocess.Popen(self.command, **kwargs)
                self.output, self.error = self.process.communicate()
                self.status = self.process.returncode
            except Exception as ex:
                self.error = traceback.format_exception_only(type(ex), ex)
                self.status = -1
        # default stdout and stderr
        if 'stdout' not in kwargs:
            kwargs['stdout'] = subprocess.PIPE
        if 'stderr' not in kwargs:
            kwargs['stderr'] = subprocess.PIPE
        # thread
        thread = threading.Thread(target=target, kwargs=kwargs)
        thread.start()
        thread.join(timeout)
        if thread.is_alive():
            self.process.terminate()
            thread.join()
        return self.status, self.output, self.error


def matchesAny(line, values):
    return line.lstrip().startswith(tuple(values))

def validateJRE(javaCmd):
    attrs = {}
    reason = ""
    operation = "validate java command: {}.".format(javaCmd)

    if os.name == "nt":
        jreInfoCmd = '"%s" %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    else:
        jreInfoCmd = '%s %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    c = Command(jreInfoCmd)
    retval, output, error = c.run(10)
    if sys.version_info[0] == 3:
        output = output.decode('utf-8')
        error = error.decode('utf-8')

    if retval != 0:
        reason = error
        logger.debug(error)
        # smartly parse the error if we can.
        for line in error:
            if line.startswith("OSError:") or line.startswith("WindowsError"):
                reason = line
        isValid = False
        logger.critical(reason)
    else:
        pairs = [ line.lstrip().split(' = ', 2) for line in error.splitlines() if matchesAny(line, C.JRE_WANTED_KEYS)]

        for pair in pairs:
            k, v = pair
            attrs[k] = v

        version = attrs.get(C.JRE_VERSION_KEY, '')
        vendor = attrs.get(C.JRE_VENDOR_KEY, '')
        vm = attrs.get(C.JRE_VM_KEY, '')

        isValid = (version == C.JRE_WANT_VERSION and (vm.startswith(C.JRE_WANT_VM_ORACLEJDK) or vm.startswith(C.JRE_WANT_VM_OPENJDK)))
        if not isValid:
            reason = {"message": "Unsupported JRE detected",
                      "jre_using": "Using %s JRE version %s, %s" % (vendor, version, vm),
                      "jre_need": "Need Oracle Corporation JRE version 1.8 or OpenSDK 1.8"}
            logger.critical(reason)

    details = [str(reason), operation]

    return isValid, " ".join(details)

if os.name == 'nt':
    JAVA_DEPENDENCIES = [os.path.join("bin", "java.exe"),
                         os.path.join("bin", "keytool.exe")]

else:
    JAVA_DEPENDENCIES = [os.path.join("bin", "java"),
                         os.path.join("bin", "keytool")]

def checkDependencies(javaHome):
    reason = ""
    for dep in JAVA_DEPENDENCIES:
        fullPath = os.path.join(javaHome, dep)
        if not os.path.exists(fullPath):
            reason = "Missing JRE dependency: %s" % fullPath
            logger.critical(reason)
            return False, reason
    return True, reason

[afx]

Hi,
I've tried the import of six and adjustment in the init function, but still get the error, am I overlooking a spot?

I really wonder how much QA went into the DB Connect 3.2 release if something like this is needed ;-(

thx
afx

[hakon_nor]

I think I missed a spot, try copying the entire code-snippet and paste it into jre_validator.py. Restart Splunk afterwards and it should work.

I've had other problems with DB-connect as well, it could be a combination of all those that creates this problem.

[afx]

Thanks, a fully copy&paste worked. There where a few more changes.

Now I run against the next brickwall:

I cannot use the datalab or SQL explorer , every attempt to execute the SQL query results in

Error in 'dbxquery' command: External search command exited unexpectedly with non-zero error code 255.

thx
afx

[hakon_nor]

Have you tried this "https://answers.splunk.com/answers/586374/splunk-db-connect-error-in-dbxquery-command-extern.html"?