topic Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before? in Splunk Search

Why does my python custom reporting command work only if I use a stats command in the pipeline before?

grundsch — Thu, 09 Mar 2017 13:26:48 GMT

I'm trying to write a new custom search command, more specifically a reporting command. I'm using the Python SDK 1.6.2.
I took the searchcommand template report.py. I removed the map function, as I want to do only a reduce operation.

If I run my custom reporting command in a search after a stats command, it works pretty well.

If I run it as the first reporting command, I get the following error:

External search command 'toxlsx' returned error code 1. Script output = "error_message=AttributeError at "/opt/splunk/etc/apps/ta-toxlsx/bin/splunklib/searchcommands/reporting_command.py", line 88 : 'function' object has no attribute 'ConfigurationSettings' "

I have the following decorator for the reporting command class:

@Configuration(run_in_preview=False)
class toxlsxCommand(ReportingCommand):

and nothing for the reduce:

 @Configuration()
 def reduce(self, events):

Any idea what's going on here?

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

koshyk — Thu, 09 Mar 2017 20:46:34 GMT

any chance to see a bit more code for reporting_command.py?

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

jkat54 — Sun, 12 Mar 2017 17:20:14 GMT

You removed the map function, but did you remove the if phase=='map': conditional statement in the prepare function?

    def map(self, records):
        """ Override this method to compute partial results.
        :param records:
        :type records:
        You must override this method, if :code:`requires_preop=True`.
        """
        return NotImplemented

    def prepare(self):

        phase = self.phase

        if phase == 'map':
            # noinspection PyUnresolvedReferences
            self._configuration = self.map.ConfigurationSettings(self)
            return

        if phase == 'reduce':
            streaming_preop = chain((self.name, 'phase="map"', str(self._options)), self.fieldnames)
            self._configuration.streaming_preop = ' '.join(streaming_preop)
            return

        raise RuntimeError('Unrecognized reporting command phase: {}'.format(json_encode_string(unicode(phase))))

This would explain why it works after a stats (phase=reduce at that point) and not before a stats because it would still possibly be in a mapping phase at that time.

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

grundsch — Mon, 13 Mar 2017 07:22:07 GMT

No, I haven't removed the condition in the prepare fuction...
The (reporting command) template show nothing about it:

#!/usr/bin/env python

import sys
from splunklib.searchcommands import \
    dispatch, ReportingCommand, Configuration, Option, validators


@Configuration()
class %(command.title())Command(ReportingCommand):
    """ %(synopsis)

    ##Syntax

    %(syntax)

    ##Description

    %(description)

    """
    @Configuration()
    def map(self, events):
        # Put your streaming preop implementation here, or remove the map method,
        # if you have no need for a streaming preop
        pass

    def reduce(self, events):
        # Put your reporting implementation
        pass

dispatch(%(command.title())Command, sys.argv, sys.stdin, sys.stdout, __name__)

and the prepare function is only found in the ReportingCommand class (splunklib/searchcommands/reporting_command.py) without info about overriding the prepare function...

I tried to remove the map section completely, or keep the return only, but it doesn't help. It triggers other errors:

with only return:

/splunklib/searchcommands/internals.py", line 519 : 'NotImplementedType' object is not iterable

(which shows that it tries to use the map() function of the base class, which returns "NotImplemented")

without the if phase=='map' section:

It triggers the exception, as the phase is not recognised.

So basically, I think I need to implement a dummy map function that does nothing. Any idea how this would look like?

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

doksu — Mon, 03 Apr 2017 12:05:42 GMT

It looks like the map function is missing a decorator, causing the AttributeError provided in the question:

@Configuration()
def map(self, records):

You're doing the right thing with the class decorator, in specifying requires_preop=False, but be sure to return NotImplemented in the map function.

Although it uses a map function, looking at the example 'sum' reporting command may also help: https://github.com/splunk/splunk-sdk-python/blob/master/examples/searchcommands_app/package/bin/sum.py

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

grundsch — Tue, 04 Apr 2017 06:22:30 GMT

I added a map function, but it leads to the same error:

External search command 'toxlsx' returned error code 1. Script output = "error_message=TypeError at "/opt/splunk-6.5.2/etc/apps/ta-toxls/bin/splunklib/searchcommands/internals.py", line 519 : 'NotImplementedType' object is not iterable "

@Configuration()
def map(self, records):
    return NotImplemented

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

grundsch — Tue, 04 Apr 2017 06:54:07 GMT

So, I added a map function that returns all records:

@Configuration()
def map(self, records):
    return records

It seems to work (no errors, my script receives all data and works as expected), but I have absolutely no idea what I'm doing.
Is that the correct way to not do anything in the map phase? (especially, what is the difference with yield() vs what I'm doing here?)

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

richielynch89 — Mon, 03 Feb 2020 16:42:05 GMT

Hi grundsch,

Would it be possible if you shared you code please?
I've been stuck on creating a reporting command for a few weeks now. I'm getting the following error even though I've tried doing what you said above:
Error: “ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Search Factory: Unknown search command 'my'.”

Any help would be much appreciated.

Thanks,
Richie.

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

grundsch — Wed, 30 Sep 2020 03:59:48 GMT

Hi Richie,
looks like you have another problem. Did you configure the command in the /local/commands.conf correctly?
i.e. something like:
[toxlsx]
filename = toxlsx.py
requires_srinfo = true
stderr_dest = message
supports_getinfo = true
supports_rawargs = true
supports_multivalues = true
run_in_preview = false

(here my command is named toxlsx, and the file name and class name muss match).
You need to restart Splunk to let it know about the new command.
If it still doesn't work, try to make the example commands in the SDK, without changes.

Good luck,
Stéphane

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

richielynch89 — Tue, 04 Feb 2020 12:30:05 GMT

My commands.conf file consists of the following:
[mycommand]
chunked=true
filename=mycommand.py

With these settings, the streaming command works just fine so I am assuming that a reporting command should work fine once I call the ReportingCommand class and appropriate methods in my own script.

Anyway, thanks for your reply. I'll keep playing around with it.

All the best,
Richie.

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

grundsch — Tue, 04 Feb 2020 12:41:25 GMT

mmmh... if your command is named "mycommand", then the class must be

@Configuration() class
mycommandCommand(ReportingCommand):

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

richielynch89 — Tue, 04 Feb 2020 13:12:38 GMT

Weird, when I use the StreamingCommand class, I do not need to append "Command" to my class name.
It works now since I have changed my class name from "MyCommand" to "MyCommandCommand".

Thank you for pointing that out Stéphane.

Regards,
Richie.

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

grundsch — Tue, 04 Feb 2020 14:27:12 GMT

yep, this whole SDK could get more documentation. For the moment, I'm staying with my previous statement:

It seems to work, but I have
absolutely no idea what I'm doing.

😄

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

richielynch89 — Tue, 04 Feb 2020 16:05:49 GMT

Yeah, I agree. its quite difficult to find clear instructions and examples for each search command type.
I found this presentation after some time of Googling. Maybe it will help shed some light in areas where you are uncertain.
https://conf.splunk.com/files/2017/slides/extending-spl-with-custom-search-commands-and-the-splunk-sdk-for-python.pdf

Also, if, like me, you run into an issue where the base search is sending a lot of events to the script (over 150k) then try processing the events in the "map" function rather than the "reduce" function.

Re: Why does my python custom reporting command work only if I use a stats command in the pipeline before?

grundsch — Tue, 04 Feb 2020 16:17:13 GMT

cool, thanks!