getinfo probe failed for external search command - a rights issue

[jhubbard74]

Issue

Splunk custom command will not work unless Splunkd is started by the root user.

If 'getinfo' is the culprit; is it due to rights/perms? and if so what rights do i need to modify on which files/directories to make it work with the splunk user?

Background

• Custom command is a python built command to calculate geometric means.
• It uses splunklib.searchcommands and imports dispatch, StreamingCommand, Configuration, Option, validators.
• paths '/opt/splunk/etc/apps/<appname>/bin/gmeans.py' and '/opt/splunk/etc/apps/<appname>/bin/splunklib/searchcommands/.py*'
• Splunk runs as user: splunk. The splunk user:group also has (775) permissions across directories and files to the gmeans.py and splunklib.
• gmeans command in Splunk is configured as "Everyone --> read - write" permissions
• have run the following command: chown -R splunk:splunk /opt/splunk
• Splunk version 6.2.0

Problem

If Splunk is started as sudo root, then gmeans.py works perfectly.
If Splunk is started as splunk, then gmeans fails with getinfo error.

commands.conf

[gmeans]
filename = gmeans.py
supports_getinfo = true
supports_rawargs = true
outputheader = true

btool command gmeans

splunk btool --debug commands list gmeans
/opt/splunk/etc/apps/<appname>/default/commands.conf [gmeans]
/opt/splunk/etc/system/default/commands.conf         changes_colorder = true
/opt/splunk/etc/system/default/commands.conf         enableheader = true
/opt/splunk/etc/apps/<appname>/default/commands.conf filename = gmeans.py
/opt/splunk/etc/system/default/commands.conf         generates_timeorder = false
/opt/splunk/etc/system/default/commands.conf         generating = false
/opt/splunk/etc/system/default/commands.conf         maxinputs = 50000
/opt/splunk/etc/system/default/commands.conf         outputheader = false
/opt/splunk/etc/system/default/commands.conf         passauth = false
/opt/splunk/etc/system/default/commands.conf         perf_warn_limit = 0
/opt/splunk/etc/system/default/commands.conf         required_fields = *
/opt/splunk/etc/system/default/commands.conf         requires_preop = false
/opt/splunk/etc/system/default/commands.conf         retainsevents = false
/opt/splunk/etc/system/default/commands.conf         streaming = false
/opt/splunk/etc/apps/<appname>/default/commands.conf supports_getinfo = true
/opt/splunk/etc/apps/<appname>/default/commands.conf supports_rawargs = true
/opt/splunk/etc/system/default/commands.conf         type = python

---

Update 6-17-2015

I installed a Brand New Splunk instance (6.2.3) installed as splunk user. I followed every piece of online documentation for creating custom streaming commands. The custom command is now in its own app. The splunklib library is located in $SPLUNK_HOME/etc/apps/(app_name)/bin/. The bin directory also holds the python script.

The attribute getinfo will not work. If I set supports_getinfo=0 the command acts like it runs, but will not return results. If I enable supports_getinfo, it returns the error. Debugging the script tells me nothing more than the error on screen.

gmeans.py

#!/usr/bin/env python

import sys
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators


# Geometric Mean Calculator
def geomean(nums):
    numbers = []
    for n in nums:
        if not isinstance(n, float):
            n = float(n)
            numbers.append(n)
        else:
            numbers.append(n)
    product = 1
    for n in numbers:
        # Prevent 0; numbers[n] should never be a 0. 
        if n < 0.25:
            n = 0.04
        product *= n
    return round(product ** (1.0 / len(numbers)), 2)


@Configuration()
class gmeansCommand(StreamingCommand):
    # Options
    fieldname = Option(
        doc=''' **Syntax:** **fieldname=***<fieldname>*
        **Description:** REQUIRED: Name of the field to hold the calcluted mean for the geometric average''',
        require=True, validate=validators.Fieldname())
    nums = Option(
        doc=''' **Syntax:** **nums=***<fieldname>*
        **Description:** REQUIRED: Name of the field that contains the list if numbers to be calculated''',
        require=True, validate=validators.Fieldname())

    def stream(self, events):
        for event in events:
            nums = []
            for n in event[self.nums].split():
                nums.append(float(n))
            event[self.fieldname] = geomean(nums)
            yield record


dispatch(gmeansCommand, sys.argv, sys.stdin, sys.stdout, __name__)

[jhubbard74 · Accepted solution]

Short Solution

The SDK that I had previously downloaded and installed was missing __init__.py and few other root python files in the splunklib folder. Once I downloaded a NEW version of the python SDK and replaced the splunklib directory (and its recursive files/directories) I was able to restart Splunk as the splunk user and it worked.

Long Answer

I modified the gmeans.py python file a bit; streamlined the code and added exception errors. This allowed the search.log to properly report that it was not finding the splunklib.searchcommands .

06-22-2015 10:27:46.297 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/etc/apps/<appname>/bin/gmeans.py __GETINFO__ nums=scores field=score': ImportError: No module named splunklib.searchcommands

Once I added the missing __init__.py to ../splunklib/ search.log then started reporting on the missing *.py files like client, binding, data ..etc.

Updated gmeans.py

import sys
from splunklib.searchcommands import \
    dispatch, StreamingCommand, Configuration, Option, validators

# Geometric Mean Calculator
def geomean(nums):
    """
        Return the geometric average of nums
        @param    list    nums    List of nums to avg
        @return   float   Geometric avg of nums
    """
    numbers = []
    for n in nums:
        if not isinstance(n, float):
            n = float(n)
            numbers.append(n)
        else:
            numbers.append(n)
    product = 1
    for n in numbers:
        if n < 0.25:
            n = 0.04
        product *= n
    return round(product ** (1.0 / len(numbers)), 2)

@Configuration()
class gmeansCommand(StreamingCommand):
    """ Computes the geometric mean of a list of numbers.
        | stats list(<field with non zero numbers>) | gmeans nums=<number list> field=<name>
    """

    # Options
    field = Option(name='field', require=True)
    nums = Option(name='nums', require=True)
    show_error = Option(name='show_error', require=False, default=False, validate=validators.Boolean())

    def stream(self, events):

        for event in events:
            try:
                event[self.field] = geomean(str(event[self.nums]).split())
            except Exception, e:
                if not self.show_error :
                    raise e            
            yield event

dispatch(gmeansCommand, sys.argv, sys.stdin, sys.stdout, __name__)

[mikemishou]

Hey, I had this same problem and it turned out that following the steps exactly as laid out online didn't work. It's not that the copy of the SDK didn't have __init__.py, it's that the instructions don't have you copy it.

When you copy splunklib/searchcommands into your app's bin directory per instructions (section "Custom Search Command Example") that causes the ImportError since there's no __init__.py copied over, that __init__.py you need is in splunklib. Therefore a more correct instruction instead of "create splunklib dir in bin/ and copy searchcommands recursively into new splunklib dir" would be "copy splunklib directory recursively into app's bin directory". That will eliminate the ImportError I was seeing which I think is the same as yours.

Sorry to resurrect a zombie but yours is the only good description and troubleshooting effort I've seen and I wanted to share for posterity. I've also submitted a correction to Splunk for the documentation I linked, which is what I think we both were following.

[DalJeanis]

upvoting @jhubbard74 answer and @mikemishou comment because you folks did a lot of work and told us what we need to know.