Splunk Search

Why does my python custom reporting command work only if I use a stats command in the pipeline before?

grundsch
Communicator

I'm trying to write a new custom search command, more specifically a reporting command. I'm using the Python SDK 1.6.2.
I took the searchcommand template report.py. I removed the map function, as I want to do only a reduce operation.

If I run my custom reporting command in a search after a stats command, it works pretty well.

If I run it as the first reporting command, I get the following error:

External search command 'toxlsx' returned error code 1. Script output = "error_message=AttributeError at "/opt/splunk/etc/apps/ta-toxlsx/bin/splunklib/searchcommands/reporting_command.py", line 88 : 'function' object has no attribute 'ConfigurationSettings' "

I have the following decorator for the reporting command class:

@Configuration(run_in_preview=False)
class toxlsxCommand(ReportingCommand):

and nothing for the reduce:

 @Configuration()
 def reduce(self, events):

Any idea what's going on here?

grundsch
Communicator

So, I added a map function that returns all records:

@Configuration()
def map(self, records):
    return records

It seems to work (no errors, my script receives all data and works as expected), but I have absolutely no idea what I'm doing.
Is that the correct way to not do anything in the map phase? (especially, what is the difference with yield() vs what I'm doing here?)

0 Karma

richielynch89
Path Finder

Hi grundsch,

Would it be possible if you shared you code please?
I've been stuck on creating a reporting command for a few weeks now. I'm getting the following error even though I've tried doing what you said above:
Error: “ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Search Factory: Unknown search command 'my'.”

Any help would be much appreciated.

Thanks,
Richie.

0 Karma

grundsch
Communicator

Hi Richie,
looks like you have another problem. Did you configure the command in the /local/commands.conf correctly?
i.e. something like:
[toxlsx]
filename = toxlsx.py
requires_srinfo = true
stderr_dest = message
supports_getinfo = true
supports_rawargs = true
supports_multivalues = true
run_in_preview = false

(here my command is named toxlsx, and the file name and class name muss match).
You need to restart Splunk to let it know about the new command.
If it still doesn't work, try to make the example commands in the SDK, without changes.

Good luck,
Stéphane

richielynch89
Path Finder

My commands.conf file consists of the following:
[mycommand]
chunked=true
filename=mycommand.py

With these settings, the streaming command works just fine so I am assuming that a reporting command should work fine once I call the ReportingCommand class and appropriate methods in my own script.

Anyway, thanks for your reply. I'll keep playing around with it.

All the best,
Richie.

0 Karma

grundsch
Communicator

mmmh... if your command is named "mycommand", then the class must be

@Configuration() class
mycommandCommand(ReportingCommand):

richielynch89
Path Finder

Weird, when I use the StreamingCommand class, I do not need to append "Command" to my class name.
It works now since I have changed my class name from "MyCommand" to "MyCommandCommand".

Thank you for pointing that out Stéphane.

Regards,
Richie.

grundsch
Communicator

yep, this whole SDK could get more documentation. For the moment, I'm staying with my previous statement:

It seems to work, but I have
absolutely no idea what I'm doing.

😄

0 Karma

richielynch89
Path Finder

Yeah, I agree. its quite difficult to find clear instructions and examples for each search command type.
I found this presentation after some time of Googling. Maybe it will help shed some light in areas where you are uncertain.
https://conf.splunk.com/files/2017/slides/extending-spl-with-custom-search-commands-and-the-splunk-s...

Also, if, like me, you run into an issue where the base search is sending a lot of events to the script (over 150k) then try processing the events in the "map" function rather than the "reduce" function.

0 Karma

grundsch
Communicator

cool, thanks!

0 Karma

doksu
SplunkTrust
SplunkTrust

It looks like the map function is missing a decorator, causing the AttributeError provided in the question:

@Configuration()
def map(self, records):

You're doing the right thing with the class decorator, in specifying requires_preop=False, but be sure to return NotImplemented in the map function.

Although it uses a map function, looking at the example 'sum' reporting command may also help: https://github.com/splunk/splunk-sdk-python/blob/master/examples/searchcommands_app/package/bin/sum....

0 Karma

grundsch
Communicator

I added a map function, but it leads to the same error:

External search command 'toxlsx' returned error code 1. Script output = "error_message=TypeError at "/opt/splunk-6.5.2/etc/apps/ta-toxls/bin/splunklib/searchcommands/internals.py", line 519 : 'NotImplementedType' object is not iterable "

@Configuration()
def map(self, records):
    return NotImplemented
0 Karma

jkat54
SplunkTrust
SplunkTrust

You removed the map function, but did you remove the if phase=='map': conditional statement in the prepare function?

    def map(self, records):
        """ Override this method to compute partial results.
        :param records:
        :type records:
        You must override this method, if :code:`requires_preop=True`.
        """
        return NotImplemented

    def prepare(self):

        phase = self.phase

        if phase == 'map':
            # noinspection PyUnresolvedReferences
            self._configuration = self.map.ConfigurationSettings(self)
            return

        if phase == 'reduce':
            streaming_preop = chain((self.name, 'phase="map"', str(self._options)), self.fieldnames)
            self._configuration.streaming_preop = ' '.join(streaming_preop)
            return

        raise RuntimeError('Unrecognized reporting command phase: {}'.format(json_encode_string(unicode(phase))))

This would explain why it works after a stats (phase=reduce at that point) and not before a stats because it would still possibly be in a mapping phase at that time.

0 Karma

grundsch
Communicator

No, I haven't removed the condition in the prepare fuction...
The (reporting command) template show nothing about it:

#!/usr/bin/env python

import sys
from splunklib.searchcommands import \
    dispatch, ReportingCommand, Configuration, Option, validators


@Configuration()
class %(command.title())Command(ReportingCommand):
    """ %(synopsis)

    ##Syntax

    %(syntax)

    ##Description

    %(description)

    """
    @Configuration()
    def map(self, events):
        # Put your streaming preop implementation here, or remove the map method,
        # if you have no need for a streaming preop
        pass

    def reduce(self, events):
        # Put your reporting implementation
        pass

dispatch(%(command.title())Command, sys.argv, sys.stdin, sys.stdout, __name__)

and the prepare function is only found in the ReportingCommand class (splunklib/searchcommands/reporting_command.py) without info about overriding the prepare function...

I tried to remove the map section completely, or keep the return only, but it doesn't help. It triggers other errors:

  • with only return:

    /splunklib/searchcommands/internals.py", line 519 : 'NotImplementedType' object is not iterable

(which shows that it tries to use the map() function of the base class, which returns "NotImplemented")

  • without the if phase=='map' section:

It triggers the exception, as the phase is not recognised.

So basically, I think I need to implement a dummy map function that does nothing. Any idea how this would look like?

0 Karma

koshyk
Super Champion

any chance to see a bit more code for reporting_command.py?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...