All Apps and Add-ons

A bytes like object is required, not string - Splunk DB Connect

Path Finder

Hi,

When I try to set the JRE Installation path(java_home) during configuration of Splunk DB connect 3.2.0 I get the following error; "A bytes like object is required, not string". I'm currently using Splunk Enterprise 8.0.1, with python 3.7 set as the default python version. This error only occurs when I use python 3.7, using python 2.7 removes the problem.

The whole problem seems to originate from how the different python versions handles strings, as described by Splunk(I dont have enough karma to post links so ill just quote it):

"In Python 2, a native, default string is a binary or "bytes" string. In Python 3, a native string is instead a unicode string. Python 3 is strict about not mixing different types of strings. Often, an explicit conversion is needed or a runtime error is produced.

In Python 2, default and binary strings shared the same type but in Python 3, you now must keep the two string types distinctly separated. Binary data should not be stored in a default string since Python 3 defaults to unicode strings, not binary."

So my question is how do I convert the string to a bytes like object?

1 Solution

Path Finder

Posting the solution to the original question given in the comments for visibility:

The solution was to change jrevalidator.py under splunkappdbconnect/bin/dbx2 to take into account the different string types. Splunk docs provided the solution, replace the file with the following code:

#! /usr/bin/env python
import os
from .dbx_logger import logger
import threading
import subprocess
import traceback
import shlex
from . import C
import sys
from six import string_types

class Command(object):
    """
    Enables to run subprocess commands in a different thread with TIMEOUT option.

    Based on jcollado's solution:
    http://stackoverflow.com/questions/1191374/subprocess-with-timeout/4825933#4825933
    """
    command = None
    process = None
    status = None
    output, error = '', ''

    def __init__(self, command):
        # string_types is a basestring() in Python 2 and str in Python 3
        if isinstance(command, string_types):
            # split the command using shell like syntax
            command = shlex.split(command)
        self.command = command


    def run(self, timeout=None, **kwargs):
        """ Run a command then return: (status, output, error). """
        def target(**kwargs):
            try:
                self.process = subprocess.Popen(self.command, **kwargs)
                self.output, self.error = self.process.communicate()
                self.status = self.process.returncode
            except Exception as ex:
                self.error = traceback.format_exception_only(type(ex), ex)
                self.status = -1
        # default stdout and stderr
        if 'stdout' not in kwargs:
            kwargs['stdout'] = subprocess.PIPE
        if 'stderr' not in kwargs:
            kwargs['stderr'] = subprocess.PIPE
        # thread
        thread = threading.Thread(target=target, kwargs=kwargs)
        thread.start()
        thread.join(timeout)
        if thread.is_alive():
            self.process.terminate()
            thread.join()
        return self.status, self.output, self.error


def matchesAny(line, values):
    return line.lstrip().startswith(tuple(values))

def validateJRE(javaCmd):
    attrs = {}
    reason = ""
    operation = "validate java command: {}.".format(javaCmd)

    if os.name == "nt":
        jreInfoCmd = '"%s" %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    else:
        jreInfoCmd = '%s %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    c = Command(jreInfoCmd)
    retval, output, error = c.run(10)
    if sys.version_info[0] == 3:
        output = output.decode('utf-8')
        error = error.decode('utf-8')

    if retval != 0:
        reason = error
        logger.debug(error)
        # smartly parse the error if we can.
        for line in error:
            if line.startswith("OSError:") or line.startswith("WindowsError"):
                reason = line
        isValid = False
        logger.critical(reason)
    else:
        pairs = [ line.lstrip().split(' = ', 2) for line in error.splitlines() if matchesAny(line, C.JRE_WANTED_KEYS)]

        for pair in pairs:
            k, v = pair
            attrs[k] = v

        version = attrs.get(C.JRE_VERSION_KEY, '')
        vendor = attrs.get(C.JRE_VENDOR_KEY, '')
        vm = attrs.get(C.JRE_VM_KEY, '')

        isValid = (version == C.JRE_WANT_VERSION and (vm.startswith(C.JRE_WANT_VM_ORACLEJDK) or vm.startswith(C.JRE_WANT_VM_OPENJDK)))
        if not isValid:
            reason = {"message": "Unsupported JRE detected",
                      "jre_using": "Using %s JRE version %s, %s" % (vendor, version, vm),
                      "jre_need": "Need Oracle Corporation JRE version 1.8 or OpenSDK 1.8"}
            logger.critical(reason)

    details = [str(reason), operation]

    return isValid, " ".join(details)

if os.name == 'nt':
    JAVA_DEPENDENCIES = [os.path.join("bin", "java.exe"),
                         os.path.join("bin", "keytool.exe")]

else:
    JAVA_DEPENDENCIES = [os.path.join("bin", "java"),
                         os.path.join("bin", "keytool")]

def checkDependencies(javaHome):
    reason = ""
    for dep in JAVA_DEPENDENCIES:
        fullPath = os.path.join(javaHome, dep)
        if not os.path.exists(fullPath):
            reason = "Missing JRE dependency: %s" % fullPath
            logger.critical(reason)
            return False, reason
    return True, reason         

View solution in original post

Path Finder

Posting the solution to the original question given in the comments for visibility:

The solution was to change jrevalidator.py under splunkappdbconnect/bin/dbx2 to take into account the different string types. Splunk docs provided the solution, replace the file with the following code:

#! /usr/bin/env python
import os
from .dbx_logger import logger
import threading
import subprocess
import traceback
import shlex
from . import C
import sys
from six import string_types

class Command(object):
    """
    Enables to run subprocess commands in a different thread with TIMEOUT option.

    Based on jcollado's solution:
    http://stackoverflow.com/questions/1191374/subprocess-with-timeout/4825933#4825933
    """
    command = None
    process = None
    status = None
    output, error = '', ''

    def __init__(self, command):
        # string_types is a basestring() in Python 2 and str in Python 3
        if isinstance(command, string_types):
            # split the command using shell like syntax
            command = shlex.split(command)
        self.command = command


    def run(self, timeout=None, **kwargs):
        """ Run a command then return: (status, output, error). """
        def target(**kwargs):
            try:
                self.process = subprocess.Popen(self.command, **kwargs)
                self.output, self.error = self.process.communicate()
                self.status = self.process.returncode
            except Exception as ex:
                self.error = traceback.format_exception_only(type(ex), ex)
                self.status = -1
        # default stdout and stderr
        if 'stdout' not in kwargs:
            kwargs['stdout'] = subprocess.PIPE
        if 'stderr' not in kwargs:
            kwargs['stderr'] = subprocess.PIPE
        # thread
        thread = threading.Thread(target=target, kwargs=kwargs)
        thread.start()
        thread.join(timeout)
        if thread.is_alive():
            self.process.terminate()
            thread.join()
        return self.status, self.output, self.error


def matchesAny(line, values):
    return line.lstrip().startswith(tuple(values))

def validateJRE(javaCmd):
    attrs = {}
    reason = ""
    operation = "validate java command: {}.".format(javaCmd)

    if os.name == "nt":
        jreInfoCmd = '"%s" %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    else:
        jreInfoCmd = '%s %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    c = Command(jreInfoCmd)
    retval, output, error = c.run(10)
    if sys.version_info[0] == 3:
        output = output.decode('utf-8')
        error = error.decode('utf-8')

    if retval != 0:
        reason = error
        logger.debug(error)
        # smartly parse the error if we can.
        for line in error:
            if line.startswith("OSError:") or line.startswith("WindowsError"):
                reason = line
        isValid = False
        logger.critical(reason)
    else:
        pairs = [ line.lstrip().split(' = ', 2) for line in error.splitlines() if matchesAny(line, C.JRE_WANTED_KEYS)]

        for pair in pairs:
            k, v = pair
            attrs[k] = v

        version = attrs.get(C.JRE_VERSION_KEY, '')
        vendor = attrs.get(C.JRE_VENDOR_KEY, '')
        vm = attrs.get(C.JRE_VM_KEY, '')

        isValid = (version == C.JRE_WANT_VERSION and (vm.startswith(C.JRE_WANT_VM_ORACLEJDK) or vm.startswith(C.JRE_WANT_VM_OPENJDK)))
        if not isValid:
            reason = {"message": "Unsupported JRE detected",
                      "jre_using": "Using %s JRE version %s, %s" % (vendor, version, vm),
                      "jre_need": "Need Oracle Corporation JRE version 1.8 or OpenSDK 1.8"}
            logger.critical(reason)

    details = [str(reason), operation]

    return isValid, " ".join(details)

if os.name == 'nt':
    JAVA_DEPENDENCIES = [os.path.join("bin", "java.exe"),
                         os.path.join("bin", "keytool.exe")]

else:
    JAVA_DEPENDENCIES = [os.path.join("bin", "java"),
                         os.path.join("bin", "keytool")]

def checkDependencies(javaHome):
    reason = ""
    for dep in JAVA_DEPENDENCIES:
        fullPath = os.path.join(javaHome, dep)
        if not os.path.exists(fullPath):
            reason = "Missing JRE dependency: %s" % fullPath
            logger.critical(reason)
            return False, reason
    return True, reason         

View solution in original post

Contributor

Hi @hakonnor,
I am also getting error in python and I tried replacing code with yours in jre
validator.py but due to which I am getting error as-<?xml version="1.0" encoding="UTF-8"?> <response> <messages> <msg type="WARN">An exception was thrown while dispatching the python script handler. </msg> </messages> </response>
and unable to load any option in splunk db connect. Kindly help is there any other things need to be change?

0 Karma

Path Finder

The error is probably caused by an indentation error in the provided script. Seems like the code sample tool used in these comments adds another space before each line of code.

0 Karma

Contributor

yes thank you

0 Karma

Contributor

Thanks, the Java option did the trick. It created the file jars/server.vmopts:

-Ddw.server.applicationConnectors[0].port=9998

And now it all works!
thx
afx

0 Karma

Path Finder

Fixed it. The solution was to change jrevalidator.py under splunkappdbconnect/bin/dbx2 to take into account the different string types. Splunk docs provided the solution, replace the file with the following code:

#! /usr/bin/env python
import os
from .dbx_logger import logger
import threading
import subprocess
import traceback
import shlex
from . import C
import sys
from six import string_types

class Command(object):
    """
    Enables to run subprocess commands in a different thread with TIMEOUT option.

    Based on jcollado's solution:
    http://stackoverflow.com/questions/1191374/subprocess-with-timeout/4825933#4825933
    """
    command = None
    process = None
    status = None
    output, error = '', ''

    def __init__(self, command):
        # string_types is a basestring() in Python 2 and str in Python 3
        if isinstance(command, string_types):
            # split the command using shell like syntax
            command = shlex.split(command)
        self.command = command


    def run(self, timeout=None, **kwargs):
        """ Run a command then return: (status, output, error). """
        def target(**kwargs):
            try:
                self.process = subprocess.Popen(self.command, **kwargs)
                self.output, self.error = self.process.communicate()
                self.status = self.process.returncode
            except Exception as ex:
                self.error = traceback.format_exception_only(type(ex), ex)
                self.status = -1
        # default stdout and stderr
        if 'stdout' not in kwargs:
            kwargs['stdout'] = subprocess.PIPE
        if 'stderr' not in kwargs:
            kwargs['stderr'] = subprocess.PIPE
        # thread
        thread = threading.Thread(target=target, kwargs=kwargs)
        thread.start()
        thread.join(timeout)
        if thread.is_alive():
            self.process.terminate()
            thread.join()
        return self.status, self.output, self.error


def matchesAny(line, values):
    return line.lstrip().startswith(tuple(values))

def validateJRE(javaCmd):
    attrs = {}
    reason = ""
    operation = "validate java command: {}.".format(javaCmd)

    if os.name == "nt":
        jreInfoCmd = '"%s" %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    else:
        jreInfoCmd = '%s %s' % (javaCmd, C.JRE_INFO_OPTIONS)
    c = Command(jreInfoCmd)
    retval, output, error = c.run(10)
    if sys.version_info[0] == 3:
        output = output.decode('utf-8')
        error = error.decode('utf-8')

    if retval != 0:
        reason = error
        logger.debug(error)
        # smartly parse the error if we can.
        for line in error:
            if line.startswith("OSError:") or line.startswith("WindowsError"):
                reason = line
        isValid = False
        logger.critical(reason)
    else:
        pairs = [ line.lstrip().split(' = ', 2) for line in error.splitlines() if matchesAny(line, C.JRE_WANTED_KEYS)]

        for pair in pairs:
            k, v = pair
            attrs[k] = v

        version = attrs.get(C.JRE_VERSION_KEY, '')
        vendor = attrs.get(C.JRE_VENDOR_KEY, '')
        vm = attrs.get(C.JRE_VM_KEY, '')

        isValid = (version == C.JRE_WANT_VERSION and (vm.startswith(C.JRE_WANT_VM_ORACLEJDK) or vm.startswith(C.JRE_WANT_VM_OPENJDK)))
        if not isValid:
            reason = {"message": "Unsupported JRE detected",
                      "jre_using": "Using %s JRE version %s, %s" % (vendor, version, vm),
                      "jre_need": "Need Oracle Corporation JRE version 1.8 or OpenSDK 1.8"}
            logger.critical(reason)

    details = [str(reason), operation]

    return isValid, " ".join(details)

if os.name == 'nt':
    JAVA_DEPENDENCIES = [os.path.join("bin", "java.exe"),
                         os.path.join("bin", "keytool.exe")]

else:
    JAVA_DEPENDENCIES = [os.path.join("bin", "java"),
                         os.path.join("bin", "keytool")]

def checkDependencies(javaHome):
    reason = ""
    for dep in JAVA_DEPENDENCIES:
        fullPath = os.path.join(javaHome, dep)
        if not os.path.exists(fullPath):
            reason = "Missing JRE dependency: %s" % fullPath
            logger.critical(reason)
            return False, reason
    return True, reason

Contributor

Hi,
I've tried the import of six and adjustment in the init function, but still get the error, am I overlooking a spot?

I really wonder how much QA went into the DB Connect 3.2 release if something like this is needed ;-(

thx
afx

0 Karma

Path Finder

I think I missed a spot, try copying the entire code-snippet and paste it into jre_validator.py. Restart Splunk afterwards and it should work.

I've had other problems with DB-connect as well, it could be a combination of all those that creates this problem.

Contributor

Thanks, a fully copy&paste worked. There where a few more changes.

Now I run against the next brickwall:

I cannot use the datalab or SQL explorer , every attempt to execute the SQL query results in

Error in 'dbxquery' command: External search command exited unexpectedly with non-zero error code 255.

thx
afx

0 Karma

Path Finder